README.md

---
license: cc-by-nc-4.0
tags:
- larql
- vindex
- qwen3
- mechanistic-interpretability
- knowledge-editing
- moe
---

# Qwen3.6-35B-A3B — LarQL Vindex v0.1

First published LarQL vindex for Qwen's Qwen3.6-35B-A3B MoE model.

A **vindex** is a transformer's weights decompiled into a queryable feature database — entity associations, circuit structure, and knowledge-editing surfaces exposed as APIs. No GPU required for most operations.

---

## What this is / What this is not

**What this IS:**
- Feature-space index for Qwen3.6-35B-A3B (35B total, 3B active, 256 experts)
- Exposes entity associations via `/v1/walk`
- Enables rank-1 knowledge edits (DELETE/INSERT) via `/v1/patch`
- Source material for `larql compile into model` → standard HuggingFace safetensors inference

**What this IS NOT:**
- A drop-in replacement for `Qwen/Qwen3.6-35B-A3B` (use that for direct generation)
- A text-generation engine — `/v1/infer` returns feature-modulated projections, not coherent completions

---

## Quickstart

```bash
# Query entity associations
curl "$LARQL_SERVICE_URL/v1/walk?prompt=Paris&layers=10-30&top=10" \
  -H "Authorization: Bearer $INTERNAL_LARQL_S2S_TOKEN"

# Apply a DELETE patch
curl -X POST "$LARQL_SERVICE_URL/v1/patches/apply" \
  -H "Authorization: Bearer $INTERNAL_LARQL_S2S_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name":"delete-example","patch":{"version":1,"base_model":"qwen3.6-35b-a3b","operations":[{"op":"delete","entity":"Paris","relation":"capital","layer":20,"feature":42}]}}'

# Compile to standard safetensors for inference on an edited model
larql compile into model \
  --vindex Divinci-AI/qwen3.6-35b-a3b-vindex \
  --output ./edited-qwen36 \
  --format safetensors
```

---

## Architecture Details

- **Architecture:** Qwen3.6 MoE (qwen3_5_moe)
- **Layers:** 40
- **Hidden size:** 2048
- **Experts:** 256 total, 8 active per token
- **MoE intermediate size:** 512 per expert
- **Source weights:** bf16 safetensors
- **Feature aggregation:** Router-weighted SVD across sampled experts, top-64 principal directions per layer

---

## Research Findings

This vindex is part of the cross-architecture study in *"Architectural Invariants of Transformer Computation: What Survives Scale, Training, and Quantization"* (arXiv forthcoming).

**Phase 1 SVD measurements (40 layers, 256 packed experts):**
- var@64 range: **0.265–0.388** (mean 0.305)
- S[0] range: 0.9–1.5 (small absolute values — tight init on 512-dim experts)
- Consistent with bf16 MoE small-expert structure; substantially higher than MXFP4 quantized models (0.032–0.066)

---

## Contents

| File | Size | Description |
|------|------|-------------|
| gate_vectors.bin | 10 MB | Aggregated gate feature directions, f16 [64 features × 2048 hidden per layer] |
| down_features.bin | 10 MB | Aggregated down-projection output directions, f16 |
| embeddings.bin | 970 MB | Token embeddings, 248,320 × 2048 (f16) |
| router_weights.bin | 41 MB | MoE router weights per layer, f16 |
| norms.bin | 324 KB | Per-layer normalization weights, f16 |
| down_meta.bin | 221 KB | Feature labels via vocab projection (top-10 tokens per feature) |
| index.json | 8 KB | Metadata: 40 layers, hidden=2048, 256 experts |
| manifest.json | 785 B | Vindex version manifest, SHA256 checksums |

**Total:** ~1.0 GB

**Note on feature aggregation:** Unlike dense-model vindexes (which store one row per FFN neuron), MoE vindexes store top-64 principal component directions aggregated across all 256 experts per layer. This keeps the artifact size tractable while preserving the dominant feature directions.

---

## Roadmap

- **Gate 3 validation** — DELETE patch suppression test pending
- **Feature clustering** — k-means over gate features (not yet included in v0.1)
- **Wikidata relation matching** — deferred to v0.2
- **Gemma 4 26B MoE vindex** — in development

---

## Built with LarQL

See [Divinci-AI/larql](https://github.com/Divinci-AI/larql) and upstream [chrishayuk/larql](https://github.com/chrishayuk/larql).

---

## Citation

```bibtex
@misc{mooring2026invariants,
  title={Architectural Invariants of Transformer Computation: What Survives Scale, Training, and Quantization},
  author={Mooring, Mike},
  year={2026},
  note={arXiv forthcoming. See https://huggingface.co/Divinci-AI/qwen3.6-35b-a3b-vindex}
}
```

## Acknowledgments

- Chris Hayuk for creating LarQL.
- Qwen team for Qwen3.6-35B-A3B.

**License:** CC-BY-NC 4.0. Academic and research use. Contact mike@divinci.ai for commercial licensing.