Instructions to use vhallac/qwen3-0.6b-nope-recal-1b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use vhallac/qwen3-0.6b-nope-recal-1b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="vhallac/qwen3-0.6b-nope-recal-1b")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("vhallac/qwen3-0.6b-nope-recal-1b")
model = AutoModelForCausalLM.from_pretrained("vhallac/qwen3-0.6b-nope-recal-1b", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use vhallac/qwen3-0.6b-nope-recal-1b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "vhallac/qwen3-0.6b-nope-recal-1b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "vhallac/qwen3-0.6b-nope-recal-1b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/vhallac/qwen3-0.6b-nope-recal-1b

SGLang

How to use vhallac/qwen3-0.6b-nope-recal-1b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "vhallac/qwen3-0.6b-nope-recal-1b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "vhallac/qwen3-0.6b-nope-recal-1b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "vhallac/qwen3-0.6b-nope-recal-1b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "vhallac/qwen3-0.6b-nope-recal-1b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use vhallac/qwen3-0.6b-nope-recal-1b with Docker Model Runner:
```
docker model run hf.co/vhallac/qwen3-0.6b-nope-recal-1b
```

Qwen3-0.6B — NoPE, recalibrated on 1B tokens

A research artifact, not a general-purpose model. This is Qwen3-0.6B with rotary position embeddings (RoPE) removed, then recalibrated on 1B tokens of FineWeb-Edu. It exists to be compared against its matched control, vhallac/qwen3-0.6b-rope-recal-1b, which received identical training with RoPE left active.

Use these two together or not at all — a single one of them in isolation answers no question.

⚠️ This model will silently misbehave if you load it normally

RoPE was disabled at runtime, by forcing the rotary embedding to return identity (cos=1, sin=0). That is not expressible in a config.json, so this repo's config is a stock Qwen3 config declaring rope_type: "default" — byte-identical to the control model's.

If you call from_pretrained and generate, RoPE will be active on a model trained without it. You will get degenerate output, no exception, and no warning.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "vhallac/qwen3-0.6b-nope-recal-1b"
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(model_id)

# REQUIRED — force rotary embeddings to identity. Without this the model is wrong.
rotary = model.model.rotary_emb
_original_forward = rotary.forward

def _identity_forward(*args, **kwargs):
    cos, sin = _original_forward(*args, **kwargs)
    return torch.ones_like(cos), torch.zeros_like(sin)

rotary.forward = _identity_forward
model.eval()

A quick way to confirm the patch took: held-out FineWeb-Edu perplexity should be ≈ 16.9. If you see a number in the thousands, the patch did not apply.

What it is for

Part of the rope-as-scaffold research program, which tested the hypothesis that RoPE is a training scaffold that can be discarded after pretraining.

The finding was that it cannot. This model is the evidence: it is what you get when you remove RoPE and recalibrate properly, and it is measurably worse than the same recipe with RoPE retained — across perplexity, local-order sensitivity, retrieval, and long-context behaviour.

Notably, the degradation is not because positional information vanishes. Probing shows position is still decodable and the emergent positional subspace substantially reconstructs the one RoPE supplied. The model reconstitutes the representation and still cannot do the job.

Results

Held-out FineWeb-Edu perplexity, fp32, 5M-token frozen eval slice, identical harness for all three:

model	CE	PPL
`Qwen/Qwen3-0.6B` (base, no extra training)	3.0819	21.80
`qwen3-0.6b-rope-recal-1b` (control, RoPE on)	2.6569	14.25
this model (RoPE removed)	2.8260	16.88

Against the base model this looks like an improvement (21.80 → 16.88). Against the correct control it is a regression: +0.169 nats / 18.4% higher perplexity, persisting after a full recalibration. The apparent gain over base is domain adaptation; the RoPE penalty is hiding underneath it.

Known limitations

Effective context ≈ 2048 tokens, despite max_position_embeddings: 40960 inherited from the base config. Beyond the recalibration window it degrades sharply: PPL ≈ 51 at 4096 and ≈ 298 at 8192, where the RoPE base model stays flat at ≈ 21.5.
Retrieval is degraded. On synthetic induction (copy a repeated random span), gain falls from the control's 12.0 nats to 11.2 at distance 512, and the gap widens with distance (12.9 → 9.4 at distance 1536 versus base).
Local-order acuity is degraded, measurably at short scrambling windows (w ≤ 8).
English-only recalibration corpus (FineWeb-Edu), while Qwen3 is multilingual + code + math. Expect domain skew relative to base beyond the eval domain.
Single seed, single recipe, 0.6B scale. Recalibrated with a cosine schedule at a 1B-token budget; a larger budget was not tested.

Training


base	`Qwen/Qwen3-0.6B` @ `c1899de289a04d12100db370d81485cdf75e47ca`
rotary	forced to identity for the entire run
corpus	`HuggingFaceFW/fineweb-edu`, `sample-10BT`, streamed in provider order
tokens	1B (1907 steps × 524,288 tokens), context 2048
optimizer	AdamW β=(0.9, 0.95), wd 0.1, grad clip 1.0
LR	1e-3 peak, 2% warmup, cosine → 10% of peak
precision	bf16
seed	0
hardware	1× H100 SXM, ~7h

training_metrics.csv and training_manifest.json are included in this repo for full provenance.

An earlier version of this checkpoint trained at LR 3e-5 plateaued at PPL ≈ 35; that learning rate was under-tuned and the run was redone at 1e-3. This repo is the corrected run.

Reproduction and full analysis

Code, specs, and lab notebook: vhallac/crockpot-experiments
The controlled comparison that produced the headline result: NOTEBOOK.md, entry 2026-07-28 — RS-amendment-2-3
Training recipe of record: RS1-spec.md §10.C

License

Apache 2.0, inherited from Qwen3-0.6B.

Downloads last month: 32

Safetensors

Model size

0.6B params

Tensor type

BF16

Model tree for vhallac/qwen3-0.6b-nope-recal-1b

Base model

Qwen/Qwen3-0.6B-Base

Finetuned

Qwen/Qwen3-0.6B