README.md

metadata

base_model: microsoft/FastContext-1.0-4B-SFT
license: mit
library_name: gguf
tags:
  - gguf
  - rocmfp4
  - qwen3
  - fastcontext
  - subagent
  - repository-exploration
  - coder
  - agentic
  - imatrix
  - strix-halo
  - amd
  - rocm
  - vulkan
language:
  - en
base_model_relation: quantized

PLUNDERSTRUCK // ROCmFP4 QUANTIZED MODEL // STRIX HALO · gfx1151

            ▗▇▇▇▇▇▇▇▖                 
           ▗█▘▝██████▖                
          ▗▛   ▝██████▆▆▆▆▆▆▆▆▆▆▅     
         ▟▛    ▗█████████████████▙▖   
   ▄▄▄▄▄▟▛    ▟████████████████████▖  
 ▗██▌    ▚▖   ▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔█▘  
▗████▖    ▜▖                    ▗█▘   
▜█████▙    ▜▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▀▀▀▀▀▜▙    
 ▜█████▙    ▝████████████▛       ▜▙   
  ▜█████▙    ▝██████████▛    ▃    ▜▙  
   ▀█████▙▖   ▝████████▘    ▟█▙    ▀▙ 
    ▝██████▖   ▝▜█████▘    ▟███▙▂▂▂▂▐█
    ▟███████▖    ▜███▘   ▗███████████▛
   ▟█████████▄    ▜▛    ▗███████████▀ 
  ▝█████▀        ▗▛    ▗██████▀▀▀▀▀▘  
    ▜██▘        ▗▛    ▟█████▛▘        
     ▜█▇▇▇▇▇▇▇▇▇█▖   ▟█████▛          
                ▝█▖ ▟█████▛           
                 ▝███████▀            

FASTCONTEXT-1.0-4B

4-BIT ROCmFP4 · QWEN3 DENSE 4B · REPO-EXPLORATION SUBAGENT · CODE-WEIGHTED IMATRIX · SINGLE AMD APU

FORMAT ROCmFP4 4-BIT	PRECISION ~4.5 BPW	ARCH QWEN3 DENSE	CONTEXT 256 K
PARAMS 4B DENSE	DRAFT NO MTP	BACKEND VULKAN0	LICENSE MIT

⚠ REQUIRES THE ROCmFP4 FORK
The custom q4_0_rocmfp4 / q4_0_rocmfp4_fast tensor types will not load in stock llama.cpp, LM Studio, or Ollama. Build/run with charlie12345/rocmfp4-llama · branch mtp-rocmfp4-strix.

NOTE // Ignore HuggingFace's auto-detected "F16"/16-bit badge — its parser can't read ROCmFP4 and mislabels the file. These are ~4.5 bpw 4-bit ROCmFP4 files; pick by filename in Files and versions.

Experimental AMD Strix Halo (gfx1151) quant of microsoft/FastContext-1.0-4B-SFT — Microsoft's repository-exploration subagent for coding agents. Instead of one model both exploring the repo and solving the task, FastContext is invoked on demand by a main agent, fires parallel read-only tool calls (READ / GLOB / GREP), and returns compact file paths + line ranges as focused context. Architecturally it's a plain Qwen3 dense 4B (Qwen3ForCausalLM, 36 layers, hidden 2560, 256K context, MIT-licensed), here in the custom ROCmFP4 4-bit format, imatrix-quantized.

01 · FILES

File	Body	Size	Pick if
…-COHERENT-embF16.gguf ★	all-dual	2.8 GB	recommended — lowest measured KL vs BF16 (§04)
…-STRIX-embF16-imatrix.gguf	fast	2.7 GB	~same fidelity, slightly smaller/faster

Both share genuine f16 embeddings (from BF16) + the code-weighted imatrix (see §04). The COHERENT build (★) puts every body tensor on the dual-scale q4_0_rocmfp4 kernel — lowest measured KL vs the BF16 reference at ~the same decode speed — vs the STRIX build's faster single-scale q4_0_rocmfp4_fast bulk. The Qwen (ChatML) chat template is baked into the GGUF — just pass --jinja.

NOTE // TIED EMBEDDINGS. FastContext has tie_word_embeddings=True, so there's no separate output head — the token-embedding tensor doubles as the lm-head. Setting --token-embedding-type f16 therefore gives an f16 embedding and f16 output head in one (no headQ6 variant needed — f16 already beats Q6 there).

02 · QUICK START

Run from the folder holding the .gguf (the Qwen ChatML template is baked in — just pass --jinja):

env HSA_OVERRIDE_GFX_VERSION=11.5.1 GGML_HIP_ENABLE_UNIFIED_MEMORY=1 \
llama-server \
  -m FastContext-1.0-4B-SFT-ROCmFP4-COHERENT-embF16.gguf \
  --alias fastcontext-4b \
  --host 0.0.0.0 \
  --port 8080 \
  -c 262144 \
  -ctk f16 \
  -ctv f16 \
  --temp 0.7 \
  --top-p 0.8 \
  --top-k 20 \
  -dev Vulkan0 \
  -ngl 999 \
  -fa on \
  -b 2048 \
  -ub 256 \
  -t 16 \
  -tb 16 \
  -cpent 256 \
  -ctxcp 32 \
  --cache-reuse 256 \
  --cache-ram 65536 \
  --jinja \
  --parallel 1 \
  --metrics \
  --no-mmap

Flag	Function
HSA_OVERRIDE_GFX_VERSION=11.5.1	treat the APU as gfx1151 (Strix Halo)
GGML_HIP_ENABLE_UNIFIED_MEMORY=1	allow use of the full 128 GB unified memory
-dev Vulkan0	run on Vulkan — fastest backend for ROCmFP4 on Strix Halo
-ngl 999 · -fa on	offload all layers · flash attention
-c 262144	context length (256K)
-b 2048 · -ub 256 · -t/-tb 16	prefill batch / micro-batch · CPU threads
-ctk f16 · -ctv f16	f16 KV cache — how we run it (cheap on a 4B); drop to q8_0/q4_0 to use less memory at deep context
-cpent · -ctxcp · --cache-reuse · --cache-ram 65536	cross-turn KV checkpointing + 64 GB resident reuse cache
--temp 0.7 --top-p 0.8 --top-k 20	Qwen3 recommended sampling (instruct/non-thinking)
--jinja --parallel 1 --metrics --no-mmap	apply baked ChatML template · single slot · metrics · weights in RAM

NOTE // No --spec-* / --spec-type draft-mtp flags — this arch has no MTP head (see §04). It's already fast on its own.

03 · USING IT AS A SUBAGENT

FastContext isn't a general chat model — it's a repository-exploration subagent meant to be called by your main coding agent, not driven directly. The intended loop: the main agent delegates "find the relevant context for X" → FastContext issues parallel read-only tool calls (READ, GLOB, GREP) → returns compact file paths + line ranges, which the main agent folds into its own context to do the actual work. The point is to keep repo-exploration tokens out of the main agent's window.

• Chat template: Qwen (ChatML) is baked into the GGUF — just pass --jinja.
• Tool calling: it emits structured READ/GLOB/GREP calls — wire those tools into your harness and use a Qwen/Hermes-style tool-call parser so they're parsed rather than printed. See the upstream model card for the exact subagent protocol + tool schema (it expects a specific invocation format).
• Sampling: temp 0.7, top-p 0.8, top-k 20 (Qwen3 instruct defaults) — already set in §02.

NOTE // It's small (4B) and fast (~68 t/s, §04) by design — a cheap, disposable explorer you can fan out in parallel next to a larger main model on the same box. The cross-turn reuse cache (--cache-reuse / --cache-ram) keeps repeated exploration over the same repo cheap.

04 · PERFORMANCE & QUALITY

DECODE · short context	~68 t/s (Vulkan / Ryzen AI Max+ 395)
SPECULATIVE DECODE	none (no MTP head)
CONTEXT	256K native (dense attention)
QUANTIZATION	COHERENT body + imatrix (measured win — below)

Recommended build = COHERENT (we measured it). Both builds use f16 tied emb/head + the same imatrix; the lever swept here is the body kernel, ranked by KL divergence vs the true BF16 on held-out code (lower = more faithful). The all-dual-scale body (COHERENT) beats the fast-body STRIX build on every metric at ~the same decode speed:

Build (imatrix + embF16, tied head)	Body	Mean KLD vs BF16 ↓	Median KLD ↓	Top-token	PPL(Q) ↓
COHERENT ★	all-dual	0.03422	0.00955	92.08%	4.192
STRIX	fast	0.03934	0.01016	91.38%	4.213

A clean sweep: COHERENT is lower on mean KLD (−13%), median KLD (−6%), RMS Δp (6.43% vs 6.95%), and perplexity (4.192 vs 4.213), and higher on same-top-token (+0.70 pp) — every metric, same direction (BF16 reference PPL 4.074). So it's the default; STRIX stays as a marginally smaller/faster fallback.

Fast on its own. ~68 t/s short-context decode on a Ryzen AI Max+ 395 (Vulkan0, measured llama-bench tg128). It's a 4B dense Qwen3 with no MTP head, so there's no speculative decoding — it doesn't need it, and at 4B it's a cheap explorer you can run several of in parallel.

NOTE // imatrix. Both builds are quantized with an importance matrix (Kalomaze groups_merged + froggeric code/technical, via froggeric/imatrix), computed on this model's BF16. We measured the COHERENT-vs-STRIX comparison above (both imatrix); we did not run a separate imatrix-vs-no-imatrix ablation on this model. Scope: the KL/PPL figures are a fidelity-vs-BF16 measurement on a held-out code slice, not an absolute coding benchmark.

05 · BUILD (REPRODUCIBLE)

# 0) convert the safetensors -> BF16 GGUF (plain qwen3 dense; no MTP, tied embeddings)
python convert_hf_to_gguf.py FastContext-1.0-4B-SFT/ --outtype bf16 --outfile FastContext-1.0-4B-SFT-BF16.gguf

# 1) imatrix on the BF16 (general+code: Kalomaze groups_merged + froggeric code/technical)
llama-imatrix -m FastContext-1.0-4B-SFT-BF16.gguf -f general+code-calib.txt -o fastcontext-4b.imatrix -c 512 -ngl 999

# 2) RECOMMENDED: COHERENT all-dual body + f16 tied emb/head (the ★ file) — lowest KL (§04).
#    tie_word_embeddings=True -> --token-embedding-type f16 also gives an f16 output head; no --output-tensor-type.
llama-quantize --token-embedding-type f16 --imatrix fastcontext-4b.imatrix \
  FastContext-1.0-4B-SFT-BF16.gguf  FastContext-1.0-4B-SFT-ROCmFP4-COHERENT-embF16.gguf  Q4_0_ROCMFP4_COHERENT

# fast-body STRIX fallback (same f16 emb + imatrix)
llama-quantize --token-embedding-type f16 --imatrix fastcontext-4b.imatrix \
  FastContext-1.0-4B-SFT-BF16.gguf  FastContext-1.0-4B-SFT-ROCmFP4-STRIX-embF16-imatrix.gguf  Q4_0_ROCMFP4_STRIX

Experimental research build for AMD Strix Halo — hardware/driver/prompt-sensitive, may not reproduce elsewhere. Not native FP4 tensor-core execution.

06 · LINEAGE & CREDITS

BASE MODEL	microsoft/FastContext-1.0-4B-SFT (MIT, Microsoft) · repository-exploration subagent · Qwen3 dense 4B (Qwen3ForCausalLM)
CALIBRATION	Kalomaze groups_merged + froggeric code/technical via froggeric/imatrix
FORMAT + RUNTIME	charlie12345/rocmfp4-llama (based on llama.cpp, MIT)

Derivative quantization — verify the base model's license before redistribution / use.