Instructions to use failspy/Meta-Llama-3-70B-Instruct-abliterated-v3.5 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use failspy/Meta-Llama-3-70B-Instruct-abliterated-v3.5 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="failspy/Meta-Llama-3-70B-Instruct-abliterated-v3.5")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("failspy/Meta-Llama-3-70B-Instruct-abliterated-v3.5")
model = AutoModelForCausalLM.from_pretrained("failspy/Meta-Llama-3-70B-Instruct-abliterated-v3.5", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use failspy/Meta-Llama-3-70B-Instruct-abliterated-v3.5 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "failspy/Meta-Llama-3-70B-Instruct-abliterated-v3.5"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "failspy/Meta-Llama-3-70B-Instruct-abliterated-v3.5",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/failspy/Meta-Llama-3-70B-Instruct-abliterated-v3.5

SGLang

How to use failspy/Meta-Llama-3-70B-Instruct-abliterated-v3.5 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "failspy/Meta-Llama-3-70B-Instruct-abliterated-v3.5" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "failspy/Meta-Llama-3-70B-Instruct-abliterated-v3.5",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "failspy/Meta-Llama-3-70B-Instruct-abliterated-v3.5" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "failspy/Meta-Llama-3-70B-Instruct-abliterated-v3.5",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use failspy/Meta-Llama-3-70B-Instruct-abliterated-v3.5 with Docker Model Runner:
```
docker model run hf.co/failspy/Meta-Llama-3-70B-Instruct-abliterated-v3.5
```

L3 8b abliterated fp16 gguf measurements for information

by Nexesenex - opened May 29, 2024

Discussion

Nexesenex

May 29, 2024

•

edited May 29, 2024

Llama 3b 8b inst Ablit v1

ARC-C: 39.13043478
ARC-E: 67.01754386
PPL-512: 11.1241

Llama 3b 8b inst Ablit v2

ARC-C: 41.47157191
ARC-E: 61.75438596
PPL-512: 8.6506

Llama 3b 8b inst Ablit v3

ARC-C: 51.17056856
ARC-E: 71.05263158
PPL-512: 8.4409

Oh, silly me, I forgot : Thank you !
And now, I'm gonna read the note ! xD

By the way, I always had the feeling that rather than finetuning such models (especially L3), which can quickly lead to low quality / debilitating overfit in order to 'drive the model", there was instead something to "remove" to expunge the models of their refusal mechanism and make them more prompt/context obedient.
Exactly what you actually did, for the reason you mentioned. Kudos for this, really.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment