---
base_model:
- meta-llama/Llama-3.1-8B
- K-intelligence/Llama-SafetyGuard-Content-Binary
library_name: peft
tags:
- mergekit
- peft
- lora
- vllm
---
# NEXS toxicity LoRA (vLLM-ready)

Rank-128 LoRA adapter for the **toxicity** domain, extracted with
[mergekit](https://github.com/arcee-ai/mergekit) from
[K-intelligence/Llama-SafetyGuard-Content-Binary](https://huggingface.co/K-intelligence/Llama-SafetyGuard-Content-Binary) against the base model
[meta-llama/Llama-3.1-8B](https://huggingface.co/meta-llama/Llama-3.1-8B), then sanitized for vLLM
serving.

## Sanitization applied

The raw mergekit extraction included full-rank `modules_to_save` tensors
(`embed_tokens`, `lm_head`, and RMSNorm layers) that vLLM's LoRA runtime does
not support. This upload contains only the pure low-rank `lora_A`/`lora_B`
weights (224 pairs: 32 layers x q/k/v/o/gate/up/down projections, bf16), with
`modules_to_save: null` in `adapter_config.json`. No `resize_token_embeddings()`
call is needed to load this adapter.

## Serving with vLLM

```bash
python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-3.1-8B \
    --enable-lora \
    --lora-modules toxicity=anjohn0077/NEXS-toxicity-lora \
    --port 8000 \
    --max-lora-rank 128 \
    --gpu-memory-utilization 0.85
```

## Evaluation (sst2)

| Variant | Accuracy |
|---|---|
| Base model | 0.6732 |
| **This LoRA on base (via vLLM)** | **0.8991** |
| Original full fine-tune | 0.8899 |

Evaluated with [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness)
against a local vLLM OpenAI-compatible endpoint:

```bash
lm_eval --model local-completions \
    --model_args model=toxicity,base_url=http://localhost:8000/v1/completions,tokenizer=meta-llama/Llama-3.1-8B,num_concurrent=10 \
    --tasks sst2 \
    --output_path results/vllm_toxicity
```