README.md

---
license: apache-2.0
base_model: badtheorylabs/BTL-4
tags:
- agentic
- tool-use
- code
- reasoning
- llama-cpp
- gguf-my-repo
pipeline_tag: text-generation
library_name: transformers
---

# gopi87/BTL-4-Q4_K_M-GGUF
This model was converted to GGUF format from [`badtheorylabs/BTL-4`](https://huggingface.co/badtheorylabs/BTL-4) using llama.cpp via the ggml.ai's [GGUF-my-repo](https://huggingface.co/spaces/ggml-org/gguf-my-repo) space.
Refer to the [original model card](https://huggingface.co/badtheorylabs/BTL-4) for more details on the model.

## Use with llama.cpp
Install llama.cpp through brew (works on Mac and Linux)
# Fix: BTL-4 GGUF fails to load — `tensor 'blk.40.attn_norm.weight' not found`

## Symptom

Loading `btl-4-q8_0.gguf` in `llama.cpp` (or `ik_llama.cpp`) fails with:

```
llama_model_load: error loading model: check_tensor_dims: tensor 'blk.40.attn_norm.weight' not found
llama_model_load_from_file_impl: failed to load model
```

This happens on **every** loader tried (mainline `llama.cpp`, `ik_llama.cpp`), including current builds that already support Qwen3.5/3.6-style NextN/MTP models.

## Root cause

The model's `general.architecture` is `qwen35moe`, and its GGUF metadata declares:

```
qwen35moe.block_count          = 41
qwen35moe.nextn_predict_layers = 1
```

This tells the loader: "40 normal transformer layers (0–39), plus 1 extra NextN/MTP speculative-decoding head layer (block 40)."

The loader walks through blocks 0–39 fine (confirmed in verbose logs — all the expected `attn_norm`, `attn_qkv`/`attn_q`/`attn_k`/`attn_v`, `ssm_*`, `ffn_*_exps` tensors load correctly). It then reaches block 40, expects to find NextN-head tensors there, and fails immediately on the first lookup.

**Verification with `gguf_dump.py` confirmed block 40 has zero tensors in the file:**

```bash
python3 gguf-py/gguf/scripts/gguf_dump.py /mnt/nvme/btl-4-q8_0.gguf 2>&1 | grep -oP "blk\.\d+" | sort -t. -k2 -n -u | tail -5
# blk.35
# blk.36
# blk.37
# blk.38
# blk.39
```

So this is **not** a loader-compatibility problem. The GGUF's metadata claims a NextN/MTP head exists, but the file was converted/quantized without ever writing the weights for it. There's nothing to recover or rename — the data simply isn't there.

## Fix

Since the NextN layer has no data anyway, correct the metadata to describe the model as it actually is: a plain 40-layer model, no MTP head. This only patches two `u32` metadata fields in place — no re-quantization, no touching the 34 GB of tensor data.

### 1. Back up the file first

```bash
cp /mnt/nvme/btl-4-q8_0.gguf /mnt/nvme/btl-4-q8_0.gguf.bak
```

### 2. Locate `gguf_set_metadata.py` in your llama.cpp checkout

```bash
find ~/llama.cpp -iname "gguf_set_metadata.py"
# typically: ~/llama.cpp/gguf-py/gguf/scripts/gguf_set_metadata.py
```

Activate the repo's venv if it has one (`source venv/bin/activate`).

### 3. Dry-run the changes to confirm the tool sees the right fields

```bash
python3 gguf-py/gguf/scripts/gguf_set_metadata.py \
  /mnt/nvme/btl-4-q8_0.gguf qwen35moe.block_count 40 --dry-run --verbose

python3 gguf-py/gguf/scripts/gguf_set_metadata.py \
  /mnt/nvme/btl-4-q8_0.gguf qwen35moe.nextn_predict_layers 0 --dry-run --verbose
```

Expected output:
```
* Preparing to change field 'qwen35moe.block_count' from 41 to 40
* Preparing to change field 'qwen35moe.nextn_predict_layers' from 1 to 0
```

### 4. Apply for real

```bash
python3 gguf-py/gguf/scripts/gguf_set_metadata.py \
  /mnt/nvme/btl-4-q8_0.gguf qwen35moe.block_count 40 --force

python3 gguf-py/gguf/scripts/gguf_set_metadata.py \
  /mnt/nvme/btl-4-q8_0.gguf qwen35moe.nextn_predict_layers 0 --force
```

Each should report `Field changed. Successful completion.`

### 5. Load normally

```bash
CUDA_VISIBLE_DEVICES=2,3,0,1 \
numactl --interleave=all \
~/llama.cpp/build/bin/llama-server \
    --model /mnt/nvme/btl-4-q8_0.gguf \
    --tensor-split 1.2,1.8,0.45,0.35 \
    --n-cpu-moe 99 \
    --ctx-size 280000 \
    --batch-size 6000 \
    --ubatch-size 6000 \
    --parallel 1 \
    --threads 42 \
    --threads-batch 42 \
    -ngl 100 \
    --host 127.0.0.1 \
    --port 8082 \
    --jinja
```

`print_info` should now show `n_layer = 40` / `n_layer_all = 40` (matching), and the server should load through to `HTTP server listening` without touching block 40 at all.

## Trade-off

This model loses the speculative-decoding speedup that a working NextN/MTP head would have given (roughly 15–70% faster decode depending on hardware, per community benchmarks on similar Qwen3.5/3.6-class models). Base inference quality is unaffected — layers 0–39 (the actual model weights) are untouched.

## If you want MTP working properly

The NextN head weights would need to be re-generated from the original `Ornith-1.0-35B` base model checkpoint (in whatever training framework Bad Theory Labs used) and the GGUF re-converted with a NextN-aware `convert_hf_to_gguf.py` that actually writes block 40's tensors — this metadata patch does not add that capability, it only stops the loader from looking for data that was never written.
```bash
brew install llama.cpp

```
Invoke the llama.cpp server or the CLI.

### CLI:
```bash
llama-cli --hf-repo gopi87/BTL-4-Q4_K_M-GGUF --hf-file btl-4-q4_k_m.gguf -p "The meaning to life and the universe is"
```

### Server:
```bash
llama-server --hf-repo gopi87/BTL-4-Q4_K_M-GGUF --hf-file btl-4-q4_k_m.gguf -c 2048
```

Note: You can also use this checkpoint directly through the [usage steps](https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#usage) listed in the Llama.cpp repo as well.

Step 1: Clone llama.cpp from GitHub.
```
git clone https://github.com/ggerganov/llama.cpp
```

Step 2: Move into the llama.cpp folder and build it with `LLAMA_CURL=1` flag along with other hardware-specific flags (for ex: LLAMA_CUDA=1 for Nvidia GPUs on Linux).
```
cd llama.cpp && LLAMA_CURL=1 make
```

Step 3: Run inference through the main binary.
```
./llama-cli --hf-repo gopi87/BTL-4-Q4_K_M-GGUF --hf-file btl-4-q4_k_m.gguf -p "The meaning to life and the universe is"
```
or
```
./llama-server --hf-repo gopi87/BTL-4-Q4_K_M-GGUF --hf-file btl-4-q4_k_m.gguf -c 2048
```