Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,6 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+wheels/tt_metal_models-0.79.0.dev20260929+qwen36p150x2.g9f6b02d-py3-none-any.whl filter=lfs diff=lfs merge=lfs -text
+wheels/ttnn-0.79.0.dev20260929+qwen36p150x2.g9f6b02d-cp312-cp312-linux_x86_64.whl filter=lfs diff=lfs merge=lfs -text
+wheels/vllm_tt_plugin-0.1.0-py3-none-any.whl filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,106 @@
+---
+license: cc-by-nc-4.0
+base_model: Altworld/Hemmingway-1
+pipeline_tag: text-generation
+tags:
+- tt-model-cache
+- blackhole
+- vllm
+- thin
+- p150x2
+- chat
+- creative-writing
+---
+
+# hemmingway-1-p300
+
+> **License: CC BY-NC 4.0. Non-commercial use only.** The weights are by Altworld (https://huggingface.co/Altworld/Hemmingway-1). Commercial use needs a separate agreement with Altworld. This package inherits that license.
+
+This is a tt-model v6 thin package that serves Altworld/Hemmingway-1 (revision `1a5f363a3dd2d1cc456c28b8abbb403b9555efaf`) on 2 Tenstorrent Blackhole chips (one P300 board, or a P150x2 mesh). Hemmingway-1 (`Altworld/Hemmingway-1`) is a 27B-parameter open-weights text model from Altworld, built on Qwen/Qwen3.8-27B for everyday writing: messages, emails and short pieces. Its upstream card says it gives you the text itself, without a preamble or a list of options. The upstream card also reports comparisons with other models. We did not reproduce them.
+
+The package was produced by an automated bring-up harness (tt-orchard), and a person reviewed the results before release.
+
+## How to serve
+
+    tt-model serve episod/hemmingway-1-p300
+
+The server exposes an OpenAI-compatible endpoint (`/v1/chat/completions` and `/v1/completions`). Context length is 262144 tokens with up to 4 sequences at a time. Speculative decoding uses the drafter `incoai/Qwen3.8-27B-DFlash2` (K=7 draft tokens per step), which was trained on base Qwen3.8-27B. It was not trained on this model.
+
+Notes that matter before you start:
+
+- This package ships no weights. `tt-model` downloads them from the upstream repository `Altworld/Hemmingway-1` with your own Hugging Face account.
+- Only greedy decoding is supported. Requests that ask for logprobs are rejected.
+- The server needs a fresh, empty tensor cache for each model. Do not share a tensor cache between Qwen3.8-based models: the cache is keyed by layer name, so a cache built for another Qwen3.8-27B model would load that model's weights under this model's name.
+- The first boot compiles kernels. Boot takes a long time (see the table below).
+
+**Turn thinking off for direct answers.** Send `"chat_template_kwargs": {"enable_thinking": false}` with each chat request. With thinking on (the server default), the model spends its token budget on reasoning and may return no visible answer. In one test with thinking on, a creative prompt used all 700 tokens on reasoning and `content` was null.
+
+## Evidence status of this card
+
+All numbers below were measured on 2026-10-04 on one box, on this package installed in its own virtual environment, with the weights and tensor cache for this model only. Each is labelled measured and gives its conditions.
+
+## Measured results (2 chips)
+
+Conditions: greedy, thinking off, tensor cache already built for this model. Speed workload: the 80 first-turn `coding` prompts of nvidia/SPEED-Bench, output cap 2048 tokens (6 of 80 hit the cap). Per-user tok/s = 1000 / mean time per output token. Short answers distort that mean, so a token-weighted figure is also given. One run per cell.
+
+| Item | Value |
+|---|---|
+| Boot, launch to `/health` OK, tensor cache present | 2025 s (33.8 min), of which 1261.7 s was the drafter and verify warm-up (alloc and compile). The verification install measured 2034.4 s. |
+| Coding, 1 user, per-user speed | 54.1 tok/s (mean 18.50 ms per token, median 13.61 ms) |
+| Coding, 1 user, token-weighted | 61.8 tok/s over 80 requests; 69.1 tok/s over the 71 requests with at least 64 output tokens |
+| Coding, 1 user, TTFT | mean 232 ms, median 185 ms |
+| Coding, 4 users, per-user speed | 28.5 tok/s (mean 35.13 ms per token, median 23.89 ms) |
+| Coding, 4 users, token-weighted | 42.2 tok/s; 42.0 tok/s over the 71 requests of at least 64 tokens |
+| Coding, 4 users, TTFT | mean 749 ms, median 728 ms |
+| Coding, aggregate | 59.3 tok/s at 1 user (24,941 output tokens in 420.5 s), 130.4 tok/s at 4 users (191.3 s) |
+
+Context sweep (`vllm bench serve`, random prompts, ignore EOS, 128 output tokens, 4 to 8 requests per cell):
+
+| Prompt tokens / users | Per user incl. TTFT | TTFT | Time per token |
+|---|---|---|---|
+| 1024 / 1 | 51.5 tok/s | 318 ms | 17.1 ms (58 tok/s decode) |
+| 16384 / 1 | 21.8 tok/s | 3788 ms | 16.3 ms (61 tok/s decode) |
+| 1024 / 4 | 30.2 tok/s (120.9 aggregate) | 969 ms | 25.7 ms |
+
+### Draft acceptance
+
+The drafter was trained on base Qwen3.8-27B. Acceptance is the mean number of draft tokens accepted per step out of 7, weighted by steps.
+
+| Workload | Accepted of 7 | Tokens committed per step |
+|---|---|---|
+| 80 coding prompts (4,644 steps) | 4.38 (62.6%) | 5.36 |
+| 10 everyday-writing prompts (356 steps) | 1.72 (24.6%) | 2.69 |
+
+Acceptance on writing prompts is much lower than on code. Speed on writing prompts was measured only as part of a 10-prompt run and not at scale. Expect lower speed on prose than the coding numbers suggest. No comparison with the base model's acceptance was run, so we cannot say whether the fine-tune lowered acceptance.
+
+### Long context retrieval
+
+One passkey in the middle of filler text, one trial per length, greedy. All were retrieved: 1063, 2075, 2944, 15783 (8.0 s) and 135567 tokens (44.5 s). The key was a 6-digit number in synthetic text. This is not a long-context quality evaluation.
+
+### Agreement with the CPU bf16 reference
+
+The chip generated 48 tokens for each of 8 prompts, and the CPU bf16 model scored each chip token given the chip's own prefix. The chip token matched at 365 of 384 positions (95.1 percent). Per prompt: 45, 42, 45, 47, 47, 46, 46, 47 of 48. The default chat template was used, so the tokens are mostly the model's thinking text. The 19 mismatches were not examined beyond their first tokens. An earlier single-prompt check gave 30 of 32 (0.9375). That is thin evidence, and the 8-prompt figure is the better one.
+
+### Writing quality check
+
+10 prompts we wrote for everyday messages and short writing (a note to a landlord, a raise request, declining a wedding invitation, a condolence message, a story opening and others), greedy, thinking off, 600-token cap. Result: 7 good, 3 partly, 0 poor. The grades were given by the AI model that ran the check, not by humans, on one answer per prompt. They support the claim that it writes the requested message directly, with no preamble or options. They say nothing about quality compared with another model, because no other model was run. The three "partly" answers had a literal placeholder ("[Manager's Name]"), an invented name ("Sarah") or a vague request.
+
+### Sample
+
+Prompt: "You are a building remembering your past. You were built in the 1930s. What do you remember?" Settings: greedy, thinking off, max_tokens 900, one sample, finished at a stop token after 830 tokens. This is not a quality measurement. Excerpt:
+
+> I remember the first family. The Kowalskis. Mrs. Kowalski hung curtains in the front window within a week of moving in, and I felt the weight of them against my glass like a small, warm hand.
+
+## What was not measured
+
+- Accuracy benchmarks (MMLU and similar). None were run.
+- A comparison with the base model Qwen3.8-27B, or with the unquantized upstream model on a GPU. None was run, so these cards make no claim that this package matches or beats either.
+- A 4-chip package. None is published here.
+- A fresh-machine pull test. Pulling this package from the Hub and booting it on a clean machine has not been done yet.
+- Speed numbers use a smaller prompt subset and a lower token cap than the earlier Qwen3.8-27B DFlash2 card, so they are not comparable with that card.
+- Sampling quality. The server accepts greedy decoding only and rejects logprobs.
+- Boot with a cold kernel cache on a fresh machine. Warm boots in these measurements took about 28 to 34 minutes, so the first boot on a new machine can be expected to take at least that long, but it was not timed.
+
+## Evidence
+
+The measurements come from the run's benchmark summary and its quality-check write-up (RESULTS-bench.md and qualitative.md), with raw result files kept by the person who ran them. The risks and open items are in the run's risk list.

base_config/config.json

@@ -0,0 +1,140 @@
+{
+    "architectures": [
+        "Qwen3_5ForConditionalGeneration"
+    ],
+    "image_token_id": 248056,
+    "language_model_only": false,
+    "model_type": "qwen3_5",
+    "text_config": {
+        "attention_bias": false,
+        "attention_dropout": 0.0,
+        "attn_output_gate": true,
+        "bos_token_id": 248044,
+        "dtype": "bfloat16",
+        "eos_token_id": 248044,
+        "full_attention_interval": 4,
+        "head_dim": 256,
+        "hidden_act": "silu",
+        "hidden_size": 5120,
+        "initializer_range": 0.02,
+        "intermediate_size": 17408,
+        "layer_types": [
+            "linear_attention",
+            "linear_attention",
+            "linear_attention",
+            "full_attention",
+            "linear_attention",
+            "linear_attention",
+            "linear_attention",
+            "full_attention",
+            "linear_attention",
+            "linear_attention",
+            "linear_attention",
+            "full_attention",
+            "linear_attention",
+            "linear_attention",
+            "linear_attention",
+            "full_attention",
+            "linear_attention",
+            "linear_attention",
+            "linear_attention",
+            "full_attention",
+            "linear_attention",
+            "linear_attention",
+            "linear_attention",
+            "full_attention",
+            "linear_attention",
+            "linear_attention",
+            "linear_attention",
+            "full_attention",
+            "linear_attention",
+            "linear_attention",
+            "linear_attention",
+            "full_attention",
+            "linear_attention",
+            "linear_attention",
+            "linear_attention",
+            "full_attention",
+            "linear_attention",
+            "linear_attention",
+            "linear_attention",
+            "full_attention",
+            "linear_attention",
+            "linear_attention",
+            "linear_attention",
+            "full_attention",
+            "linear_attention",
+            "linear_attention",
+            "linear_attention",
+            "full_attention",
+            "linear_attention",
+            "linear_attention",
+            "linear_attention",
+            "full_attention",
+            "linear_attention",
+            "linear_attention",
+            "linear_attention",
+            "full_attention",
+            "linear_attention",
+            "linear_attention",
+            "linear_attention",
+            "full_attention",
+            "linear_attention",
+            "linear_attention",
+            "linear_attention",
+            "full_attention"
+        ],
+        "linear_conv_kernel_dim": 4,
+        "linear_key_head_dim": 128,
+        "linear_num_key_heads": 16,
+        "linear_num_value_heads": 48,
+        "linear_value_head_dim": 128,
+        "mamba_ssm_dtype": "float32",
+        "max_position_embeddings": 262144,
+        "model_type": "qwen3_5_text",
+        "mtp_num_hidden_layers": 1,
+        "mtp_use_dedicated_embeddings": false,
+        "num_attention_heads": 24,
+        "num_hidden_layers": 64,
+        "num_key_value_heads": 4,
+        "output_gate_type": "swish",
+        "pad_token_id": null,
+        "partial_rotary_factor": 0.25,
+        "rms_norm_eps": 1e-06,
+        "rope_parameters": {
+            "mrope_interleaved": true,
+            "mrope_section": [
+                11,
+                11,
+                10
+            ],
+            "partial_rotary_factor": 0.25,
+            "rope_theta": 10000000,
+            "rope_type": "default"
+        },
+        "tie_word_embeddings": false,
+        "use_cache": true,
+        "vocab_size": 248320
+    },
+    "tie_word_embeddings": false,
+    "transformers_version": "5.8.0.dev0",
+    "video_token_id": 248057,
+    "vision_config": {
+        "deepstack_visual_indexes": [],
+        "depth": 27,
+        "hidden_act": "gelu_pytorch_tanh",
+        "hidden_size": 1152,
+        "in_channels": 3,
+        "initializer_range": 0.02,
+        "intermediate_size": 4304,
+        "model_type": "qwen3_5",
+        "num_heads": 16,
+        "num_position_embeddings": 2304,
+        "out_hidden_size": 5120,
+        "patch_size": 16,
+        "spatial_merge_size": 2,
+        "temporal_patch_size": 2
+    },
+    "vision_end_token_id": 248054,
+    "vision_start_token_id": 248053
+}

base_config/preprocessor_config.json

@@ -0,0 +1,21 @@
+{
+    "size": {
+        "longest_edge": 16777216,
+        "shortest_edge": 65536
+    },
+    "patch_size": 16,
+    "temporal_patch_size": 2,
+    "merge_size": 2,
+    "image_mean": [
+        0.5,
+        0.5,
+        0.5
+    ],
+    "image_std": [
+        0.5,
+        0.5,
+        0.5
+    ],
+    "processor_class": "Qwen3VLProcessor",
+    "image_processor_type": "Qwen2VLImageProcessorFast"
+}

base_config/video_preprocessor_config.json

@@ -0,0 +1,21 @@
+{
+    "size": {
+        "longest_edge": 25165824,
+        "shortest_edge": 4096
+    },
+    "patch_size": 16,
+    "temporal_patch_size": 2,
+    "merge_size": 2,
+    "image_mean": [
+        0.5,
+        0.5,
+        0.5
+    ],
+    "image_std": [
+        0.5,
+        0.5,
+        0.5
+    ],
+    "processor_class": "Qwen3VLProcessor",
+    "video_processor_type": "Qwen3VLVideoProcessor"
+}

install.sh

@@ -0,0 +1,36 @@
+#!/usr/bin/env bash
+# Install this self-contained TT model package into an isolated, reproducible venv (via uv).
+# Usage: ./install.sh [venv-path]   (default: ./venv)
+# Deps: v6 thin: ttnn/tt-metal-models (index) + empty-target vLLM + plugin/ops wheels (by path)
+#
+# HERMETIC INSTALL: everything the model needs to SERVE ends up UNDER this folder — the pinned
+# interpreter (in .python/), the venv (with package contents copied in), and at serve time the
+# caches/weights (run.sh points HF_HOME/TT_CACHE_PATH/... here). After this runs, serving depends
+# on nothing outside the folder except the TT device + system libc. Only THIS install step reaches
+# the network (to fetch the interpreter and, unless --vendor-deps, the pip deps).
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+VENV="${1:-$HERE/venv}"
+PYVER="3.12"
+
+# uv gives us a pinned interpreter + deterministic installs, independent of the host Python.
+if ! command -v uv >/dev/null 2>&1; then
+  export UV_INSTALL_DIR="$HERE/.uv"
+  curl -LsSf https://astral.sh/uv/install.sh | sh >/dev/null 2>&1
+  export PATH="$HERE/.uv:$PATH"
+fi
+
+# Keep the pinned interpreter INSIDE the bundle (not in uv's global ~/.local store), so the venv's
+# python resolves within the folder wall. python-build-standalone (what uv provisions) is
+# relocatable, so a --relocatable venv built against it stays self-contained.
+export UV_PYTHON_INSTALL_DIR="$HERE/.python"
+uv python install "$PYVER"
+uv venv --relocatable --python "$PYVER" "$VENV"
+uv pip install --python "$VENV/bin/python" --link-mode=copy --find-links "$HERE/wheels" --extra-index-url https://download.pytorch.org/whl/cpu -r "$HERE/requirements.txt"
+VLLM_COMMON="$(mktemp)"
+curl -fsSL "https://raw.githubusercontent.com/vllm-project/vllm/v0.26.0/requirements/common.txt" -o "$VLLM_COMMON"
+uv pip install --python "$VENV/bin/python" --link-mode=copy --override "$HERE/vllm-overrides.txt" -r "$VLLM_COMMON"
+rm -f "$VLLM_COMMON"
+VLLM_TARGET_DEVICE=empty uv pip install --python "$VENV/bin/python" --link-mode=copy --no-deps --no-binary vllm vllm==0.26.0
+uv pip install --python "$VENV/bin/python" --link-mode=copy --find-links "$HERE/wheels" "$HERE/wheels/vllm_tt_plugin-0.1.0-py3-none-any.whl"
+echo "installed into $VENV (python $PYVER, interpreter under $HERE/.python)"

model.py

@@ -0,0 +1,6 @@
+"""Thin-bundle runner shim for Qwen3.8-27B on Blackhole (DFlash2 speculative decoding).
+
+The model code itself ships in the `tt-metal-models` wheel (models.demos.blackhole.qwen36); this module only
+re-exports the vLLM adapter class so `model:Qwen36DFlashForCausalLM` resolves with PYTHONPATH set to the bundle dir.
+"""
+from models.demos.blackhole.qwen36.tt.qwen36_vllm_dflash import Qwen36DFlashForCausalLM  # noqa: F401

prepare_model_dir.py

@@ -0,0 +1,83 @@
+#!/usr/bin/env python3
+"""Build model-dir/ for this bundle before vLLM starts. run.sh runs it with the bundle's python.
+
+Written by tt-orchard (stage 7) for a model whose weights are a fine-tune of a supported model.
+The model code in this bundle is registered for the base model's architecture, so vLLM must see
+the base model's config files. The tokenizer and weights must be the fine-tune's. This script
+builds model-dir/ next to itself from both:
+
+- COPIED from base_config/ (shipped in the bundle): every file there, for example config.json.
+- LINKED from the fine-tune's Hugging Face snapshot (absolute links to the resolved files):
+  tokenizer.json, tokenizer_config.json, chat_template.jinja, generation_config.json,
+  model.safetensors.index.json and every *.safetensors file. tokenizer.json and at least one
+  *.safetensors file are required.
+
+The snapshot is the manifest's `weights.repo_id` at `weights.revision` (which must be a pinned
+40-character commit) under $HF_HUB_CACHE or $HF_HOME/hub (run.sh sets HF_HOME). When it is missing
+and HF_HUB_OFFLINE is not set, it is downloaded with huggingface_hub from the bundle's venv. When it
+is missing and HF_HUB_OFFLINE is set, the script exits 2 and names it.
+
+model-dir/.weights records "<repo>@<revision>". A model-dir without that file was not built here,
+and the script exits 2 without touching it. Exit 0 means model-dir is ready.
+"""
+from __future__ import annotations
+
+import json
+import os
+import re
+import shutil
+import sys
+from pathlib import Path
+
+HERE = Path(__file__).resolve().parent
+LINKED = ("tokenizer.json", "tokenizer_config.json", "chat_template.jinja", "generation_config.json",
+          "model.safetensors.index.json")
+MARKER = ".weights"
+
+
+def fail(message: str) -> None:
+    print(f"prepare_model_dir: {message}", file=sys.stderr)
+    sys.exit(2)
+
+
+def snapshot(repo: str, rev: str) -> Path:
+    hub = os.environ.get("HF_HUB_CACHE") or os.path.join(
+        os.environ.get("HF_HOME") or os.path.expanduser("~/.cache/huggingface"), "hub")
+    org, name = repo.split("/", 1)
+    snap = Path(hub) / f"models--{org}--{name}" / "snapshots" / rev
+    if snap.is_dir():
+        return snap
+    if os.environ.get("HF_HUB_OFFLINE", "").lower() in ("1", "true", "yes", "on"):
+        fail(f"the weights snapshot {repo}@{rev} is not in {hub} and HF_HUB_OFFLINE is set. "
+             f"Download it first: hf download {repo} --revision {rev}")
+    from huggingface_hub import snapshot_download      # in the bundle's venv
+    return Path(snapshot_download(repo_id=repo, revision=rev))
+
+
+def main() -> int:
+    weights = json.loads((HERE / "tt_kernel_manifest.json").read_text(encoding="utf-8"))["weights"]
+    repo, rev = weights["repo_id"], weights.get("revision") or ""
+    if not re.fullmatch(r"[0-9a-f]{40}", rev):
+        fail(f"the manifest's weights revision {rev!r} is not a 40-character commit")
+    md = HERE / "model-dir"
+    if md.is_symlink() or (md.exists() and not (md / MARKER).is_file()):
+        fail(f"{md} exists and was not built by this script; move it aside")
+    snap = snapshot(repo, rev)
+    names = list(LINKED) + sorted(p.name for p in snap.glob("*.safetensors"))
+    present = [n for n in names if (snap / n).exists()]
+    if "tokenizer.json" not in present or not any(n.endswith(".safetensors") for n in present):
+        fail(f"{snap} needs tokenizer.json and at least one *.safetensors file")
+    if md.exists():
+        shutil.rmtree(md)
+    md.mkdir()
+    for f in sorted((HERE / "base_config").iterdir()):
+        shutil.copyfile(f, md / f.name)
+    for n in present:
+        os.symlink(os.path.realpath(snap / n), md / n)
+    (md / MARKER).write_text(f"{repo}@{rev}", encoding="utf-8")
+    print(f"prepare_model_dir: {md} holds the base config and {len(present)} files of {repo}@{rev}")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

requirements.txt

@@ -0,0 +1,5 @@
+--extra-index-url https://download.pytorch.org/whl/cpu
+ttnn==0.79.0.dev20260929+qwen36p150x2.g9f6b02d
+tt-metal-models==0.79.0.dev20260929+qwen36p150x2.g9f6b02d
+transformers==5.17.0
+tokenizers==0.23.2

run.sh

@@ -0,0 +1,87 @@
+#!/usr/bin/env bash
+# Serve this model on TT hardware. Assumes ./install.sh has been run.
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+VENV="${VENV:-$HERE/venv}"
+PYBIN="$VENV/bin/python"
+
+# Locate ttnn WITHOUT importing it — importing loads _ttnn.so, which is exactly what needs the
+# LD_PRELOAD below (chicken-and-egg). find_spec resolves the path without executing the module.
+TTNN_DIR="$("$PYBIN" -c 'import importlib.util,os;print(os.path.dirname(importlib.util.find_spec("ttnn").origin))')"
+# _ttnncpp.so lives in ttnn.libs/ for an auditwheel-repaired (portable) wheel, or build/lib/ for a
+# raw one; preload it to avoid the glibc "static TLS block" error on late dlopen.
+# Prefer the auditwheel-vendored copy in *.libs/ (that's the one _ttnn.so actually loads via
+# RPATH); fall back to build/lib for a raw (unrepaired) wheel — e.g. the plain `ttnn` PyPI wheel
+# a v6 thin bundle installs, which isn't auditwheel-repaired at all.
+# `|| true` on each probe: under `set -e -o pipefail`, a `ls <no-match> | head -1` pipe fails
+# (pipefail surfaces ls's nonzero exit) and set -e would kill the script on THIS line, before the
+# fallback below — or the deliberate `:?` error a few lines down — ever runs.
+LD_PRELOAD="$(ls "$TTNN_DIR"/../*.libs/_ttnncpp*.so 2>/dev/null | head -1)" || true
+[ -n "$LD_PRELOAD" ] || LD_PRELOAD="$(ls "$TTNN_DIR"/build/lib/_ttnncpp*.so 2>/dev/null | head -1)" || true
+export LD_PRELOAD="${LD_PRELOAD:?could not locate _ttnncpp.so in the ttnn install}"
+export TT_METAL_HOME="$TTNN_DIR"
+# EXTRA_MODELS_DIR is a PARENT of per-model bundle folders; the plugin scans its children for
+# each vllm_metadata.json (so the metadata lives in vllm_models/<model>/, not the bundle root).
+export EXTRA_MODELS_DIR="$HERE/vllm_models"
+export TT_VLLM_BUILTIN_MODELS=0
+# Do NOT set VLLM_PLUGINS: it is an ALLOW-LIST — setting it suppresses the vllm.general_plugins
+# group, so the model's tool/reasoning-parser overrides would silently not load. The TT platform
+# + model registry load via entry points without it.
+export PYTHONPATH="$HERE:${PYTHONPATH:-}"   # resolves the adapter/model imports
+export MESH_DEVICE="${MESH_DEVICE:-P150x2}"
+NCHIPS=2
+if [ -z "${TT_METAL_VISIBLE_DEVICES:-}" ]; then
+  if [ -n "${TT_VISIBLE_DEVICES:-}" ]; then
+    IFS=, read -ra _GRANT <<< "$TT_VISIBLE_DEVICES"
+    if [ "${#_GRANT[@]}" -lt "$NCHIPS" ]; then
+      echo "run.sh: this model needs $NCHIPS chip(s) but TT_VISIBLE_DEVICES grants ${#_GRANT[@]} ($TT_VISIBLE_DEVICES)" >&2
+      exit 1
+    fi
+    TT_VISIBLE_DEVICES="$(IFS=,; echo "${_GRANT[*]:0:$NCHIPS}")"
+    export TT_VISIBLE_DEVICES
+    TT_METAL_VISIBLE_DEVICES="0,1"
+  else
+    TT_METAL_VISIBLE_DEVICES="0,1"
+  fi
+fi
+export TT_METAL_VISIBLE_DEVICES
+# HERMETIC RUNTIME: keep every cache/home INSIDE the folder wall, so serving writes and reads
+# nothing outside it (the ttnn tensor cache even DEFAULTS to a hard-coded /mnt/... path upstream —
+# a classic other-machine leak we must override). Each is overridable if the operator sets it.
+export HF_HOME="${HF_HOME:-$HERE/.hf}"                  # HF weights + hub cache
+export TT_CACHE_PATH="${TT_CACHE_PATH:-$HERE/.tt_cache}"    # ttnn weight/tensor cache
+export TT_CACHE_HOME="${TT_CACHE_HOME:-$HERE/.tt_cache}"    # override upstream's /mnt/... default
+export XDG_CACHE_HOME="${XDG_CACHE_HOME:-$HERE/.cache}"     # generic catch-all (triton, etc.)
+export TRITON_CACHE_DIR="${TRITON_CACHE_DIR:-$HERE/.cache/triton}"
+export TORCHINDUCTOR_CACHE_DIR="${TORCHINDUCTOR_CACHE_DIR:-$HERE/.cache/inductor}"
+export HF_MODEL="$HERE/model-dir"
+export MODEL_WEIGHTS_DIR="$HERE/model-dir"
+export TT_MODEL_WEIGHTS_REVISION="${TT_MODEL_WEIGHTS_REVISION:-1a5f363a3dd2d1cc456c28b8abbb403b9555efaf}"
+export ARCH_NAME="blackhole"
+export QWEN36_SKIP_VISION="1"
+export QWEN_SDPA_BF8="1"
+export TORCHDYNAMO_DISABLE="1"
+export TT_QWEN35_TEXT_VER="qwen36_blackhole"
+export VLLM_RPC_TIMEOUT="900000"
+export VLLM_CONFIGURE_LOGGING="1"
+export QWEN36_DRAFTER="dflash2"
+export DFLASH_WEIGHTS="incoai/Qwen3.8-27B-DFlash2@dedf8df68adfb1afeaf7b7480c0a0243108177b4"
+export QWEN36_DFLASH_TP="1"
+export QWEN36_DFLASH_BLOCK="8"
+export QWEN36_DFLASH_FOLD_SEED="1"
+export QWEN36_DFLASH_SERVE_BLOCK="32"
+export QWEN36_PREFILL_BUCKET_TRACE="0"
+export TT_METAL_PINNED_MEMORY_CACHE_LIMIT_BYTES="0"
+export QWEN36_GDN_SPEC_FUSED="1"
+export QWEN36_MAX_TOKENS_ALL_USERS="262144"
+CMD=("$PYBIN" -m vllm.entrypoints.openai.api_server --model "$HERE/model-dir" --max_num_seqs 4 --block_size 64 --max_model_len 262144 --additional-config '{"tt": {"l1_small_size": 24576, "fabric_config": "FABRIC_1D", "trace_region_size": 1073741824, "sample_on_device_mode": "decode_only"}}' --max-num-batched-tokens 65536 --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning_parser qwen3 --no-async-scheduling "$@")
+# TT_MODEL_PRINT=1 (set by `tt-model serve --print`) echoes the fully-resolved command+env
+if [ "${TT_MODEL_PRINT:-0}" = "1" ]; then
+  printf 'LD_PRELOAD=%s TT_METAL_HOME=%s EXTRA_MODELS_DIR=%s MESH_DEVICE=%s HF_MODEL=%s
+  %s
+' \
+    "$LD_PRELOAD" "$TT_METAL_HOME" "$EXTRA_MODELS_DIR" "$MESH_DEVICE" "${HF_MODEL:-}" "${CMD[*]}"
+  exit 0
+fi
+"$PYBIN" "$HERE/prepare_model_dir.py"
+exec "${CMD[@]}"

tt_kernel_manifest.json

@@ -0,0 +1,80 @@
+{
+  "schema_version": "6",
+  "name": "hemmingway-1-p300",
+  "tt_metal_version": "0.79.0.dev20260929+qwen36p150x2.g9f6b02d",
+  "arch": "blackhole",
+  "device_count": 2,
+  "producer": {
+    "tt_kernel_version": "0.1.0",
+    "created_at": "2026-10-04T01:18:16.031583+00:00",
+    "hostname": "redacted"
+  },
+  "weights": {
+    "repo_id": "Altworld/Hemmingway-1",
+    "revision": "1a5f363a3dd2d1cc456c28b8abbb403b9555efaf",
+    "allow_patterns": null,
+    "ignore_patterns": null,
+    "repo_type": "model"
+  },
+  "mesh": {
+    "devices": 2,
+    "topology": "P150x2",
+    "fabric": null
+  },
+  "entrypoint": {
+    "cls": "models.demos.blackhole.qwen36.tt.qwen36_vllm_dflash:Qwen36DFlashForCausalLM",
+    "arch_name": "Qwen3_5ForConditionalGeneration"
+  },
+  "resources": {
+    "max_model_len": 262144,
+    "max_num_seqs": 4,
+    "block_size": 64,
+    "trace_region_bytes": null,
+    "extra_args": [],
+    "command_override": {}
+  },
+  "capabilities": null,
+  "env": {
+    "ARCH_NAME": "blackhole",
+    "QWEN36_SKIP_VISION": "1",
+    "QWEN_SDPA_BF8": "1",
+    "TORCHDYNAMO_DISABLE": "1",
+    "TT_QWEN35_TEXT_VER": "qwen36_blackhole",
+    "VLLM_RPC_TIMEOUT": "900000",
+    "VLLM_CONFIGURE_LOGGING": "1",
+    "QWEN36_DRAFTER": "dflash2",
+    "DFLASH_WEIGHTS": "incoai/Qwen3.8-27B-DFlash2@dedf8df68adfb1afeaf7b7480c0a0243108177b4",
+    "QWEN36_DFLASH_TP": "1",
+    "QWEN36_DFLASH_BLOCK": "8",
+    "QWEN36_DFLASH_FOLD_SEED": "1",
+    "QWEN36_DFLASH_SERVE_BLOCK": "32",
+    "QWEN36_PREFILL_BUCKET_TRACE": "0",
+    "TT_METAL_PINNED_MEMORY_CACHE_LIMIT_BYTES": "0",
+    "QWEN36_GDN_SPEC_FUSED": "1",
+    "QWEN36_MAX_TOKENS_ALL_USERS": "262144"
+  },
+  "bundled": null,
+  "deps": {
+    "python": "3.12",
+    "requirements": "requirements.txt",
+    "wheels": [
+      "wheels/vllm_tt_plugin-0.1.0-py3-none-any.whl"
+    ],
+    "wheels_dir": "wheels",
+    "models_wheels": [
+      "wheels/ttnn-0.79.0.dev20260929+qwen36p150x2.g9f6b02d-cp312-cp312-linux_x86_64.whl",
+      "wheels/tt_metal_models-0.79.0.dev20260929+qwen36p150x2.g9f6b02d-py3-none-any.whl"
+    ],
+    "vllm": {
+      "version": "0.26.0",
+      "target_device": "empty",
+      "overrides": "vllm-overrides.txt",
+      "common_requirements": null,
+      "wheel": null
+    },
+    "model_dir": ".",
+    "kind": "vllm",
+    "app": null
+  },
+  "container": null
+}

vllm-overrides.txt

@@ -0,0 +1,6 @@
+# vLLM dependency overrides for the empty-target (TT) build — see tenstorrent/vllm-tt-plugin.
+# ttnn needs numpy<2; vLLM's common.txt wants opencv-python-headless>=4.13 which needs numpy>=2.
+# Pin opencv to the last numpy-1 release (its video path is unused by TT models) and hold numpy<2
+# (fixed by the tt-metal/ttnn env this runs inside). Revisit on a vLLM bump or if a model gains video.
+opencv-python-headless==4.11.0.86
+numpy>=1.24.4,<2

vllm_models/hemmingway-1-p300/vllm_metadata.json

@@ -0,0 +1,4 @@
+{
+  "arch": "Qwen3_5ForConditionalGeneration",
+  "main_class": "models.demos.blackhole.qwen36.tt.qwen36_vllm_dflash:Qwen36DFlashForCausalLM"
+}

wheels/tt_metal_models-0.79.0.dev20260929+qwen36p150x2.g9f6b02d-py3-none-any.whl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4528111a0415b9f5d0f5468b7645a1bf7d6d719cd4c5ff527fbc4434893b3390
+size 9544741

wheels/ttnn-0.79.0.dev20260929+qwen36p150x2.g9f6b02d-cp312-cp312-linux_x86_64.whl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:502ad84809ad51e70cfcab1f79372319ddbff1acf984ca9652cddf8bbd586cdd
+size 89713784

wheels/vllm_tt_plugin-0.1.0-py3-none-any.whl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2fa641a876be2523d4cd7bd2f363643412e55060fc6b34b4d78ec42d0cd826ba
+size 322371