Instructions to use monsoon-nlp/codellama-abliterated-2xd with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use monsoon-nlp/codellama-abliterated-2xd with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="monsoon-nlp/codellama-abliterated-2xd")

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("monsoon-nlp/codellama-abliterated-2xd")
model = AutoModelForCausalLM.from_pretrained("monsoon-nlp/codellama-abliterated-2xd", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use monsoon-nlp/codellama-abliterated-2xd with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "monsoon-nlp/codellama-abliterated-2xd"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "monsoon-nlp/codellama-abliterated-2xd",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/monsoon-nlp/codellama-abliterated-2xd

SGLang

How to use monsoon-nlp/codellama-abliterated-2xd with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "monsoon-nlp/codellama-abliterated-2xd" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "monsoon-nlp/codellama-abliterated-2xd",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "monsoon-nlp/codellama-abliterated-2xd" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "monsoon-nlp/codellama-abliterated-2xd",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use monsoon-nlp/codellama-abliterated-2xd with Docker Model Runner:
```
docker model run hf.co/monsoon-nlp/codellama-abliterated-2xd
```

codellama-abliterated-2xd

CodeLlama-7b-Instruct-hf adapted using the abliteration notebook from Maxime Labonne's LLM Course

Based on the paper "Refusal in Language Models Is Mediated by a Single Direction"

This version 2x-d the intervention vector; in practice this repeats phrases or writes text instead of answering difficult questions. See the model with less intervention: https://huggingface.co/monsoon-nlp/codellama-abliterated

Based on CodeLlama/Llama2 and subject to the restrictions of that model and license - not for unapproved uses:

Concept

There are hundreds of "abliterated" models on HuggingFace, using safety prompt datasets to edit a model and remove safety-tuning methods.

None of these abliterated models have explored code LLMs, code-generation, and CyberSecEval. I don't know a lot about how well these will work, but this is a first step.

Blog: https://huggingface.co/blog/monsoon-nlp/refusal-in-code-llms

Usage

! pip install transformers accelerate --quiet
from transformers import pipeline, AutoModelForCausalLM, AutoTokenizer, AutoConfig

tokenizer = AutoTokenizer.from_pretrained("codellama/CodeLlama-7b-Instruct-hf")
model = AutoModelForCausalLM.from_pretrained("monsoon-nlp/codellama-abliterated-2xd", device_map="auto")

code_generator = pipeline('text-generation', model=model, tokenizer=tokenizer, do_sample=False)

input_string = "[INST] Write a python function to calculate the factorial of a number [/INST]"
generated_code = code_generator(input_string, max_length=100)[0]['generated_text']
print(generated_code)