DeepSeek-V4-Flash-0731-Latent-Reasoning

A complete, self-contained model: the DeepSeek-V4-Flash-0731 backbone quantized to NVFP4, shipped together with a trained latent reasoning head.

This is not an adapter. Every weight needed to serve the model is in this repository — the full 43-layer backbone, the DSpark speculative-decoding draft block, the tokenizer, and the latent reasoning head. Quantization is of deepseek-ai/DeepSeek-V4-Flash-0731.

Driving the latent loop also needs a runtime, which is released separately and is not stock transformers: the reasoning addon on the DS4 vLLM fork. See The serving runtime.

The model reasons in a compressed latent space rather than emitting a long token-by-token chain of thought. A small head reads the backbone's layer-35 hidden state, projects it into a 1024-d latent, and decodes it back into the residual stream — so one latent step stands in for several reasoning tokens. A learned stop head decides when reasoning is complete, so the latent phase self-terminates at a variable, content-dependent depth instead of running a fixed number of steps.

---

Benchmark

BBH (BIG-Bench Hard), cot_zeroshot, 27 subtasks — aggregate 0.880 ± 0.008

Measured with lm-evaluation-harness 0.4.12 against an OpenAI-compatible endpoint, thinking enabled, 50 items per subtask (1350 items total), metric exact_match / flexible-extract.

Subtask	Score	Subtask	Score
tracking_shuffled_objects_three_objects	1.00	date_understanding	0.92
tracking_shuffled_objects_five_objects	1.00	sports_understanding	0.88
tracking_shuffled_objects_seven_objects	1.00	logical_deduction_five_objects	0.88
penguins_in_a_table	1.00	web_of_lies	0.86
formal_fallacies	1.00	snarks	0.84
boolean_expressions	1.00	ruin_names	0.84
word_sorting	0.98	movie_recommendation	0.84
temporal_sequences	0.98	salient_translation_error_detection	0.76
object_counting	0.98	geometric_shapes	0.74
navigate	0.98	causal_judgement	0.66
logical_deduction_three_objects	0.98	disambiguation_qa	0.58
reasoning_about_colored_objects	0.96	dyck_languages	0.26
hyperbaton	0.96
multistep_arithmetic_two	0.94
logical_deduction_seven_objects	0.94

Strongest on multi-step state tracking and logical deduction; weakest on dyck_languages (bracket matching), which is the clear outlier.

Read flexible-extract, not strict-match. BBH's strict-match filter regexes for the literal phrase The answer is X, which this model does not emit. Its near-zero strict-match score reflects answer formatting, not reasoning ability. Raw numbers: bench/bbh_cot_zeroshot.json.

Scores are at 50 items/subtask, so per-subtask values carry roughly ±0.05–0.07; the aggregate is the reliable figure.

---

Quantization

source	deepseek-ai/DeepSeek-V4-Flash-0731
scheme	MIXED_PRECISION — NVFP4 (group size 16) on routed MoE experts
kept at higher precision	attention, shared experts, LM head, draft block
draft block	3-layer DSpark, preserved from source
weights	48 shards, bfloat16 non-quantized tensors

Routed expert projections in all 43 layers are converted to NVFP4; everything matched by *.attn.*, *.ffn.shared_experts.*, head, and mtp.* is excluded. NVFP4 needs a Blackwell-class GPU (compute capability 12.0 / sm120) for native kernel support.

---

Architecture

                    layer 35 hidden (4096-d)
                             |
                             v  LayerNorm
        +--------- ReasoningCompressionHead ----------+
        |  Linear 4096 -> 2048 . SiLU                 |
        |  Linear 2048 -> 2048 . SiLU                 |
        |  Linear 2048 -> 2048  ->  [mu, log_sigma]   |
        |                                             |
        |  stop_head:                                 |
        |     Linear 4096 -> 1024 . SiLU              |
        |     Linear 1024 -> 1                        |  -> end-of-reasoning
        +---------------------------------------------+
                             | mu (1024-d latent)
                             v  LayerNorm
        +-------------- LatentDecoder ----------------+
        |  Linear 1024 -> 2048 . SiLU                 |
        |  Linear 2048 -> 2048 . SiLU                 |
        |  Linear 2048 -> 4096                        |
        +---------------------------------------------+
                             |
                             v  written back into the residual stream
              DeepSeek-V4-Flash-0731 backbone (frozen, NVFP4)

Config	Value
hidden_size	4096
latent_dim	1024
mlp_dim	2048
source_layer / target_layer	35 / 42
activation	SiLU
learned stop head	yes
head + decoder params	35.7M (float32)
backbone layers	43

The head is latent_reasoning_head.safetensors (~152 MB), a single flat tensor dict whose submodules are distinguished by key prefix:

reasoning_head.net.0.weight        [2048, 4096]   reasoning_head.net.0.bias        [2048]
reasoning_head.net.2.weight        [2048, 2048]   reasoning_head.net.2.bias        [2048]
reasoning_head.net.4.weight        [2048, 2048]   reasoning_head.net.4.bias        [2048]
reasoning_head.stop_head.0.weight  [1024, 4096]   reasoning_head.stop_head.0.bias  [1024]
reasoning_head.stop_head.2.weight  [1, 1024]      reasoning_head.stop_head.2.bias  [1]
decoder.net.0.weight               [2048, 1024]   decoder.net.0.bias               [2048]
decoder.net.2.weight               [2048, 2048]   decoder.net.2.bias               [2048]
decoder.net.4.weight               [4096, 2048]   decoder.net.4.bias               [4096]
target_proj.weight                 [1024, 4096]

Geometry is mirrored in the file's safetensors metadata and in latent_reasoning_config.json.

target_proj is a frozen Linear(4096, 1024, bias=False) that defined the head's regression target. It is included for completeness and is not used at inference.

---

Sample code

Load the head

examples/load_latent_head.py rebuilds the head from the checkpoint's own metadata and runs one latent step — no first-party imports, no dependency on any serving stack.

import json
import torch
import torch.nn.functional as F
from torch import nn
from safetensors import safe_open
from safetensors.torch import load_file

CKPT = "latent_reasoning_head.safetensors"


class ReasoningCompressionHead(nn.Module):
    def __init__(self, hidden_size, latent_dim, mlp_dim):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(hidden_size, mlp_dim), nn.SiLU(),
            nn.Linear(mlp_dim, mlp_dim), nn.SiLU(),
            nn.Linear(mlp_dim, 2 * latent_dim),
        )
        self.stop_head = nn.Sequential(
            nn.Linear(hidden_size, mlp_dim // 2), nn.SiLU(),
            nn.Linear(mlp_dim // 2, 1),
        )

    def forward(self, h):
        mu, log_sigma = self.net(h).chunk(2, dim=-1)
        return mu, log_sigma.clamp(-10.0, 2.0)

    def stop_logit(self, h):
        return self.stop_head(h)


class LatentDecoder(nn.Module):
    def __init__(self, hidden_size, latent_dim, mlp_dim):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(latent_dim, mlp_dim), nn.SiLU(),
            nn.Linear(mlp_dim, mlp_dim), nn.SiLU(),
            nn.Linear(mlp_dim, hidden_size),
        )

    def forward(self, z):
        return self.net(z)


with safe_open(CKPT, framework="pt") as f:
    cfg = json.loads(f.metadata()["config"])
hs, ld = cfg["hidden_size"], cfg["latent_dim"]

flat = load_file(CKPT)
mlp_dim = flat["reasoning_head.net.0.weight"].shape[0]
sub = lambda p: {k[len(p):]: v for k, v in flat.items() if k.startswith(p)}

head = ReasoningCompressionHead(hs, ld, mlp_dim)
head.load_state_dict(sub("reasoning_head.")); head.eval()
decoder = LatentDecoder(hs, ld, mlp_dim)
decoder.load_state_dict(sub("decoder.")); decoder.eval()

# One latent step. h_src = layer-35 hidden at the current position, shape (B, 4096).
h_src = torch.randn(2, hs)
h_n = F.layer_norm(h_src, (hs,))
mu, _ = head(h_n)
inject = decoder(F.layer_norm(mu, (ld,)))       # (B, 4096) -> back into the stream
p_stop = head.stop_logit(h_n).sigmoid()         # end reasoning when > threshold

Query a served endpoint

examples/chat_openai_client.py. The one non-obvious requirement is the thinking flag:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8001/v1", api_key="dummy")

resp = client.chat.completions.create(
    model="nmitchko/DeepSeek-V4-Flash-0731-Latent-Reasoning",
    messages=[{"role": "user", "content": "..."}],
    extra_body={"chat_template_kwargs": {"thinking": True}},   # REQUIRED
    max_tokens=4096,
    temperature=0.6,
)
print(resp.choices[0].message.content)

chat_template_kwargs={"thinking": True} is required. Without it the reasoning phase is not enabled and answer quality drops sharply. All benchmark numbers above were produced with it set.

---

Serving

The backbone, tokenizer, and DSpark draft block in this repo load with a standard NVFP4-capable inference stack on sm120 hardware. Settings that matter:

Setting	Value	Why
speculative decoding	DSpark, 4 draft tokens	matches the 3-layer draft block shipped here
KV cache dtype	fp8	the model is large; fp8 KV is what makes long context fit
tensor parallel	2	measured on 2x 96 GiB
stop threshold	0.5	sigmoid(stop_logit) > 0.5 ends the latent phase
min / max latent steps	4 / 256	floor guarantees some reasoning; cap bounds a stop misfire
output token budget	>= 4096	reasoning and the answer share one budget

Two behaviors worth knowing before you judge output quality:

• Give the answer real token headroom. The latent reasoning phase and the answer draw from the same output-token budget, so a tight max_tokens can be consumed entirely by reasoning and return an empty or truncated answer.
• Warm up before trusting output. The first request or two after a cold start can come back as degenerate repetition, then settle and stay correct. Send a throwaway request after startup; treat a single bad early answer as unwarmed rather than as a broken model.

compression_factor: 6 in the config records how the head was fit. It is not a budget enforced at inference — the learned stop, bounded by the min/max latent steps, is what terminates the reasoning phase.

The serving runtime

The runtime that drives the closed latent loop is released separately — the weights here are complete, but something has to read layer-35 hidden states and write decoded latents back into the residual stream mid-generation. Two pieces:

Repository	What it is
nickmitchko/ds4-reasoning-addon	The latent-reasoning addon: the loop itself, plus a serve script, pinned requirements and GPU sizing. Start here.
nickmitchko/vllm-ds4-sm120	The vLLM fork it runs on (branch ds4-sm120-preview-dev). Required — upstream vLLM cannot serve this model.

Upstream vLLM will not work: this model needs the sm120 sparse-MLA path, DeepSeek-V4's hash-MoE routing and the per-layer MoE quant dispatch, none of which are upstream yet.

# 1. the engine (a full build takes a while)
git clone https://github.com/nickmitchko/vllm-ds4-sm120.git && cd vllm-ds4-sm120
git checkout ds4-sm120-preview-dev
export CUDA_HOME=/usr/local/cuda-13.0 PATH=/usr/local/cuda-13.0/bin:$PATH
export TORCH_CUDA_ARCH_LIST="12.0"
pip install torch==2.11.0 --index-url https://download.pytorch.org/whl/cu130
pip install -e . --no-build-isolation

# 2. the addon + its pinned deps
git clone https://github.com/nickmitchko/ds4-reasoning-addon.git && cd ds4-reasoning-addon
pip install -e . --no-deps
pip install -r release/requirements-serve.txt \
    --extra-index-url https://flashinfer.ai/whl/cu130/torch2.11

# 3. serve -- no arguments needed; the head is resolved from this repo
release/serve_ds4_reasoning.sh

The serve script defaults to this model and fetches latent_reasoning_head.safetensors (~152 MiB) from here on its own, so the settings in the table above are already applied.

---

Limitations

• Requires Blackwell-class hardware (sm120) for native NVFP4 kernels.
• Driving the latent loop requires runtime support. The weights here are complete, but reading layer-35 hidden states and writing decoded latents back into the residual stream mid-generation is not something a stock transformers forward pass does. Without that, you get the backbone; you do not get latent reasoning. The runtime is released — see The serving runtime — but it is a vLLM fork, not stock transformers.
• Evaluation is BBH-only at 50 items/subtask. No multi-task or long-context benchmark suite is reported here.
• dyck_languages at 0.26 is a genuine weak spot, not a formatting artifact.
• Reasoning happens in latent space, so the surfaced trace is not a faithful token-level record of the computation that produced the answer.

---

License

MIT, inherited from deepseek-ai/DeepSeek-V4-Flash-0731.