Text Generation
GGUF
llama.cpp
agent
coding
reasoning
tool-use
function-calling
quantized
cuda
metal
conversational
Instructions to use badtheorylabs/BTL-3-Compact with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use badtheorylabs/BTL-3-Compact with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf badtheorylabs/BTL-3-Compact # Run inference directly in the terminal: llama cli -hf badtheorylabs/BTL-3-Compact
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf badtheorylabs/BTL-3-Compact # Run inference directly in the terminal: llama cli -hf badtheorylabs/BTL-3-Compact
Use pre-built binary
# Download pre-built binary from: # https://github.com/ggerganov/llama.cpp/releases # Start a local OpenAI-compatible server with a web UI: ./llama-server -hf badtheorylabs/BTL-3-Compact # Run inference directly in the terminal: ./llama-cli -hf badtheorylabs/BTL-3-Compact
Build from source code
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-server llama-cli # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama-server -hf badtheorylabs/BTL-3-Compact # Run inference directly in the terminal: ./build/bin/llama-cli -hf badtheorylabs/BTL-3-Compact
Use Docker
docker model run hf.co/badtheorylabs/BTL-3-Compact
- LM Studio
- Jan
- vLLM
How to use badtheorylabs/BTL-3-Compact with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "badtheorylabs/BTL-3-Compact" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "badtheorylabs/BTL-3-Compact", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/badtheorylabs/BTL-3-Compact
- Ollama
How to use badtheorylabs/BTL-3-Compact with Ollama:
ollama run hf.co/badtheorylabs/BTL-3-Compact
- Unsloth Desktop
- Pi
How to use badtheorylabs/BTL-3-Compact with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf badtheorylabs/BTL-3-Compact
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "badtheorylabs/BTL-3-Compact" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use badtheorylabs/BTL-3-Compact with Docker Model Runner:
docker model run hf.co/badtheorylabs/BTL-3-Compact
- Lemonade
How to use badtheorylabs/BTL-3-Compact with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull badtheorylabs/BTL-3-Compact
Run and chat with the model
lemonade run user.BTL-3-Compact-{{QUANT_TAG}}List all available models
lemonade list
- Hermes Agent
How to use badtheorylabs/BTL-3-Compact with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf badtheorylabs/BTL-3-Compact
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default badtheorylabs/BTL-3-Compact
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use badtheorylabs/BTL-3-Compact with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf badtheorylabs/BTL-3-Compact
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "badtheorylabs/BTL-3-Compact" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Publish BTL-3 Compact package metadata
Browse files- RELEASE_MANIFEST.json +55 -0
- SHA256SUMS +67 -0
- docs/btl-3-compact-gguf-exporter.md +69 -0
- docs/launch-btl3-compact.md +153 -0
- docs/launch-btl3-cuda.md +161 -0
- docs/patched-ollama-and-lmstudio.md +234 -0
- evidence/BTL-3-Compact-AVQ2.report.json +45 -0
- evidence/compact-validation.md +122 -0
- evidence/native-load-report.json +23 -0
- evidence/rtx-pro-6000-speed-2026-07-19.md +59 -0
- integrations/btl3-native/README.md +29 -0
- integrations/btl3-native/manifest.json +7 -0
- integrations/btl3-native/model.yaml +17 -0
- integrations/btl3-native/package-lock.json +221 -0
- integrations/btl3-native/package.json +17 -0
- integrations/btl3-native/run-local +11 -0
- integrations/btl3-native/src/config.ts +53 -0
- integrations/btl3-native/src/generator.ts +185 -0
- integrations/btl3-native/src/index.ts +1 -0
- integrations/btl3-native/src/nativeRunner.ts +120 -0
- integrations/btl3-native/tsconfig.json +14 -0
- integrations/ollama/Modelfile +5 -0
- licenses/LICENSE +177 -0
- licenses/LICENSE.runtime +22 -0
- licenses/THIRD_PARTY_NOTICES.md +30 -0
- tools/install_consumer_bundle.py +148 -0
RELEASE_MANIFEST.json
ADDED
|
@@ -0,0 +1,55 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"schema_version": 1,
|
| 3 |
+
"release": "BTL-3 Compact",
|
| 4 |
+
"checkpoint": "RL-0013",
|
| 5 |
+
"architecture": "Qwen3.6-27B",
|
| 6 |
+
"model": {
|
| 7 |
+
"path": "model/BTL-3-Compact-AVQ2.gguf",
|
| 8 |
+
"bytes": 8392369600,
|
| 9 |
+
"sha256": "2ddf9527620a17a2a6739d184a7096c45712092e6589128792ec6254e94dc30c"
|
| 10 |
+
},
|
| 11 |
+
"runtimes": {
|
| 12 |
+
"supported": [
|
| 13 |
+
{
|
| 14 |
+
"name": "BTL-3-Compact-macos-arm64",
|
| 15 |
+
"path": "runtimes/supported/BTL-3-Compact-macos-arm64",
|
| 16 |
+
"bundle": "BTL-3 Compact macOS arm64",
|
| 17 |
+
"status": "verified native runtime"
|
| 18 |
+
}
|
| 19 |
+
],
|
| 20 |
+
"preview": [
|
| 21 |
+
{
|
| 22 |
+
"name": "BTL-3-Compact-linux-arm64-cuda",
|
| 23 |
+
"path": "runtimes/preview/BTL-3-Compact-linux-arm64-cuda",
|
| 24 |
+
"bundle": "BTL-3 Compact Linux arm64 CUDA (DGX Spark)",
|
| 25 |
+
"status": "cross-compiled; NVIDIA runtime conformance pending"
|
| 26 |
+
}
|
| 27 |
+
]
|
| 28 |
+
},
|
| 29 |
+
"integrations": [
|
| 30 |
+
"integrations/btl3-native",
|
| 31 |
+
"integrations/ollama"
|
| 32 |
+
],
|
| 33 |
+
"documentation": [
|
| 34 |
+
"docs/launch-btl3-compact.md",
|
| 35 |
+
"docs/launch-btl3-cuda.md",
|
| 36 |
+
"docs/patched-ollama-and-lmstudio.md",
|
| 37 |
+
"docs/btl-3-compact-gguf-exporter.md"
|
| 38 |
+
],
|
| 39 |
+
"evidence": [
|
| 40 |
+
"evidence/BTL-3-Compact-AVQ2.report.json",
|
| 41 |
+
"evidence/native-load-report.json",
|
| 42 |
+
"evidence/rtx-pro-6000-speed-2026-07-19.md",
|
| 43 |
+
"evidence/compact-validation.md"
|
| 44 |
+
],
|
| 45 |
+
"tools": [
|
| 46 |
+
"tools/install_consumer_bundle.py"
|
| 47 |
+
],
|
| 48 |
+
"licenses": [
|
| 49 |
+
"licenses/LICENSE",
|
| 50 |
+
"licenses/LICENSE.runtime",
|
| 51 |
+
"licenses/THIRD_PARTY_NOTICES.md"
|
| 52 |
+
],
|
| 53 |
+
"stock_ollama_compatible": false,
|
| 54 |
+
"stock_lm_studio_engine_compatible": false
|
| 55 |
+
}
|
SHA256SUMS
ADDED
|
@@ -0,0 +1,67 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
60abfe4b90cb4be69481bbe4df2619ed0732f9222c65b9368ec881a099cc6835 README.md
|
| 2 |
+
87e8ab7ed85be38e270830b157061a99af89fb85f51eb12c411e6dcecf5d88e8 RELEASE_MANIFEST.json
|
| 3 |
+
7c3d8064baf55779579b48c58fa083e0bda2a639c857b4bde56bab4008d28002 docs/btl-3-compact-gguf-exporter.md
|
| 4 |
+
d1f83f8a4652b7d07417067e964af864aaebf559bb9b0c76a54a0c357c3ffe86 docs/launch-btl3-compact.md
|
| 5 |
+
ad9f659ec619d9ba3ce07ea864d994f312947d8e7c47abf52b1df659f557811b docs/launch-btl3-cuda.md
|
| 6 |
+
47611c1f38ce2151446883eff93f3cf0f87e3ce41714efdc9324d5c344a377b5 docs/patched-ollama-and-lmstudio.md
|
| 7 |
+
9425bf13a02b906a7f0af92f94cb3970a8c4b30190d6920908c3235111a4aade evidence/BTL-3-Compact-AVQ2.report.json
|
| 8 |
+
c7441c024c90d293b1e95a8eb97d5e3ce6d007d218f6e53aab443a0e9c96d71b evidence/compact-validation.md
|
| 9 |
+
46140e71a8d13348dea6735c039aab8201d9bbbaaff438362b50c4f4f384e02c evidence/native-load-report.json
|
| 10 |
+
ba9f674a698aa8344596eff3ee698eaf67727222181a0463bc393b3467dddde4 evidence/rtx-pro-6000-speed-2026-07-19.md
|
| 11 |
+
63016e04a641e0eb1741aff3591fcda65cb57c2bdd9d96d4bfa7c7d65a6456b5 integrations/btl3-native/README.md
|
| 12 |
+
78cb0785678d836893fe0ddce58b4c7ff34beec050158ff25f479abc02d562a1 integrations/btl3-native/manifest.json
|
| 13 |
+
0dda87a718d77f5d3d4d47017b37b1261c29349f6d9cede2b639309fda5a348d integrations/btl3-native/model.yaml
|
| 14 |
+
3b2125b24cffabc49b490ca8824656309670f02f0d1a4f7aea2699cf6d47668a integrations/btl3-native/package-lock.json
|
| 15 |
+
f47708fb76e59e85c44a0b61928793d53290880d5e6b93e40926c21f1e223811 integrations/btl3-native/package.json
|
| 16 |
+
47c2254ff3cae7b23631dc01a84deefa25940befa8852d08ac2cc1bbef53abd5 integrations/btl3-native/run-local
|
| 17 |
+
6e2a1a2c5838436c7762ee6e0d879737acbc4dac6dd9d7a3a0f03ac4d7bfe72d integrations/btl3-native/src/config.ts
|
| 18 |
+
51a8605efddaa8fafd708be26e0ac5cd6b12002b948b2a8f4c4c27fe452f33d7 integrations/btl3-native/src/generator.ts
|
| 19 |
+
2840145e096cdea78b8323ba29258b1dd80c8ee7ab66e72a4550fe77a5010a76 integrations/btl3-native/src/index.ts
|
| 20 |
+
6ec5c47c5991089837db57630255e05bb9eda81c63592f962a6a5387d716a2c0 integrations/btl3-native/src/nativeRunner.ts
|
| 21 |
+
d74511d134d59f432bc5108a3a0449c29e7a75346511cceda10575fb8ac084e7 integrations/btl3-native/tsconfig.json
|
| 22 |
+
6492d9473f6f6c7c6126cb11259324e2081fdc2c3dfbcad5fa22534d3680de32 integrations/ollama/Modelfile
|
| 23 |
+
0d542e0c8804e39aa7f37eb00da5a762149dc682d7829451287e11b938e94594 licenses/LICENSE
|
| 24 |
+
ef85e198bda0c18a42650ea1a576bffdc3f8749d2f8dcec739b9be34efeebb2d licenses/LICENSE.runtime
|
| 25 |
+
87cfc2262dbf499daff47db1339293198fba61431f623fd7226004d6714d58c8 licenses/THIRD_PARTY_NOTICES.md
|
| 26 |
+
2ddf9527620a17a2a6739d184a7096c45712092e6589128792ec6254e94dc30c model/BTL-3-Compact-AVQ2.gguf
|
| 27 |
+
94f29bbed6a22c35b992c5c6ebf0e7c92f13b836b90f36f461c9cf2f0f1d010d runtimes/preview/BTL-3-Compact-linux-arm64-cuda/LICENSE.llama.cpp
|
| 28 |
+
043658e4900b04a80fb9d4247c4fe8b04f5adc3706325996dc176b202f9baa24 runtimes/preview/BTL-3-Compact-linux-arm64-cuda/bin/btl3-server
|
| 29 |
+
328388c97746e3781f0fe8520ec73aa4bb8ad55978c2a62385964fff33e711d0 runtimes/preview/BTL-3-Compact-linux-arm64-cuda/bundle-manifest.json
|
| 30 |
+
7761d260387b95fa13b7ec101453abca45ef972d3e6ccc8bcf54251e1ac0470b runtimes/preview/BTL-3-Compact-linux-arm64-cuda/lib/libcublas.so.13
|
| 31 |
+
3839e12c3b69ed8eb732e5d5f8e749403a582a9d706f779017c785152d9e62f9 runtimes/preview/BTL-3-Compact-linux-arm64-cuda/lib/libcublasLt.so.13
|
| 32 |
+
7bdba2b5b08cbdc85203c41cc94598adedb1bcfea7cb574ca693ac73599e4e63 runtimes/preview/BTL-3-Compact-linux-arm64-cuda/lib/libcudart.so.13
|
| 33 |
+
b18971c1a35490194c8aa1666a57494e6324064155017add217f0b20ade657ba runtimes/preview/BTL-3-Compact-linux-arm64-cuda/lib/libggml-base.so.0
|
| 34 |
+
b18971c1a35490194c8aa1666a57494e6324064155017add217f0b20ade657ba runtimes/preview/BTL-3-Compact-linux-arm64-cuda/lib/libggml-base.so.0.13.1
|
| 35 |
+
b61975da167b511286b5df2d6cc384d3ba9eb78d559927d784352d420a4d59db runtimes/preview/BTL-3-Compact-linux-arm64-cuda/lib/libggml-cpu.so
|
| 36 |
+
9edb3b2a88fe59fe6d33d9e6a70f8bede4df68b3acfc5e00a63c53e34163985c runtimes/preview/BTL-3-Compact-linux-arm64-cuda/lib/libggml-cuda.so
|
| 37 |
+
3ab1d08c26865f5ff7ed753818cd84bc1fa9438e633eae7e3a53b633794973f5 runtimes/preview/BTL-3-Compact-linux-arm64-cuda/lib/libggml.so.0
|
| 38 |
+
3ab1d08c26865f5ff7ed753818cd84bc1fa9438e633eae7e3a53b633794973f5 runtimes/preview/BTL-3-Compact-linux-arm64-cuda/lib/libggml.so.0.13.1
|
| 39 |
+
cde7a5d92f37a641b8962fd2a4a516edfadb19a53b426ab392d10995053df7c0 runtimes/preview/BTL-3-Compact-linux-arm64-cuda/lib/libllama-cli-impl.so
|
| 40 |
+
a154d3ee1da598eb55ff0ecfbdcd6898fa449a1dbac7c2afcf22263eac7abae6 runtimes/preview/BTL-3-Compact-linux-arm64-cuda/lib/libllama-common.so.0
|
| 41 |
+
a154d3ee1da598eb55ff0ecfbdcd6898fa449a1dbac7c2afcf22263eac7abae6 runtimes/preview/BTL-3-Compact-linux-arm64-cuda/lib/libllama-common.so.0.0.0
|
| 42 |
+
44c74d0f766c9f0fa1bf0482eabf12927b9a5615e18c348c8cd19e3c2c952f25 runtimes/preview/BTL-3-Compact-linux-arm64-cuda/lib/libllama-server-impl.so
|
| 43 |
+
78addbbc0eb03da6208eab469e8e17a989e47f3e3c99674aa3203fd658694021 runtimes/preview/BTL-3-Compact-linux-arm64-cuda/lib/libllama.so.0
|
| 44 |
+
78addbbc0eb03da6208eab469e8e17a989e47f3e3c99674aa3203fd658694021 runtimes/preview/BTL-3-Compact-linux-arm64-cuda/lib/libllama.so.0.0.0
|
| 45 |
+
60a791c271d779f4f11321a13dd380424ca20909f530b035ea3d568ab2b87178 runtimes/preview/BTL-3-Compact-linux-arm64-cuda/lib/libmtmd.so.0
|
| 46 |
+
60a791c271d779f4f11321a13dd380424ca20909f530b035ea3d568ab2b87178 runtimes/preview/BTL-3-Compact-linux-arm64-cuda/lib/libmtmd.so.0.0.0
|
| 47 |
+
5e1c37e59fb2cd37acd8df0fe87a5e1f7a74a496af4ae278012757d84a1b1939 runtimes/preview/BTL-3-Compact-linux-arm64-cuda/libexec/llama-cli
|
| 48 |
+
b67f9d10eaa85a500180ddecf6ecdf0e9c100b0bdf4c10e32b0bf657ca50e494 runtimes/preview/BTL-3-Compact-linux-arm64-cuda/libexec/llama-server
|
| 49 |
+
7d5450cb2d142651b8afa315b5f238efc805dad827d91ba367d8516bc9d49e7a runtimes/supported/BTL-3-Compact-macos-arm64/LICENSE.OpenSSL
|
| 50 |
+
94f29bbed6a22c35b992c5c6ebf0e7c92f13b836b90f36f461c9cf2f0f1d010d runtimes/supported/BTL-3-Compact-macos-arm64/LICENSE.llama.cpp
|
| 51 |
+
de4350f194d6f393bd3ea1065b024d7f92fea0a8f2e435f6557dc0c6696b3a74 runtimes/supported/BTL-3-Compact-macos-arm64/bin/btl3-ollama-bridge
|
| 52 |
+
f14fe48ab3d5f0662f7ffe524cd6db8bc4d9bb2d892e54147c1e290cd2532d75 runtimes/supported/BTL-3-Compact-macos-arm64/bin/btl3-server
|
| 53 |
+
3ac935decfa3c67faaa210dee3b07417611cc1abcf13c5fcf5850fac3cbd399e runtimes/supported/BTL-3-Compact-macos-arm64/bundle-manifest.json
|
| 54 |
+
1f6ea0bde7b3d4ea462b047e12905c8b554b111d2808deb33fc3add67b4d0895 runtimes/supported/BTL-3-Compact-macos-arm64/lib/libcrypto.3.dylib
|
| 55 |
+
a5edff95c1a5f9c6fd3072ea65cb8b75e44fb92bbb1b35901ec43fe62cbc6eff runtimes/supported/BTL-3-Compact-macos-arm64/lib/libggml-base.0.13.1.dylib
|
| 56 |
+
14c8604076154609c9767a936953f7c2e419099a509e129629467329b5d21aad runtimes/supported/BTL-3-Compact-macos-arm64/lib/libggml-blas.0.13.1.dylib
|
| 57 |
+
9c4a1850b1c55891fb0c9f050c3ba42951b724cff9da8e9ba666fc79dfbd2014 runtimes/supported/BTL-3-Compact-macos-arm64/lib/libggml-cpu.0.13.1.dylib
|
| 58 |
+
60827547a100f266ba2003d71ff875e69bd228c6ba0882c478eb70591236971f runtimes/supported/BTL-3-Compact-macos-arm64/lib/libggml-metal.0.13.1.dylib
|
| 59 |
+
0cee0d7d52b6146f0806931e3dd298b4ad7d14ca7dee09877550710856031d72 runtimes/supported/BTL-3-Compact-macos-arm64/lib/libggml.0.13.1.dylib
|
| 60 |
+
62ad74a419305415bb82949c4e2aab62011544905af956aa35a5b6de37212cd0 runtimes/supported/BTL-3-Compact-macos-arm64/lib/libllama-common.0.0.9596.dylib
|
| 61 |
+
5fa01307a6369735e27caeba88f7c918f48bb3618db77049733b62a7cca072c2 runtimes/supported/BTL-3-Compact-macos-arm64/lib/libllama-server-impl.dylib
|
| 62 |
+
803311ec14455f2d0c48a79501d10134ec7d7a91c135363ca066be0b77ba19e0 runtimes/supported/BTL-3-Compact-macos-arm64/lib/libllama.0.0.9596.dylib
|
| 63 |
+
1a50c4f282eaaa080918262324f0095eb189857eb05d3604931c52e6039ea818 runtimes/supported/BTL-3-Compact-macos-arm64/lib/libmtmd.0.0.9596.dylib
|
| 64 |
+
7bc4c3a34a7f6b90eb3715de8ccb76bf0262882b9c3fb5dc2bdb13a30b0fe0bf runtimes/supported/BTL-3-Compact-macos-arm64/lib/libssl.3.dylib
|
| 65 |
+
822a7679463a66d994b14f1d6fcab80f08fae0c3839d83d08fb8778b78f685f8 runtimes/supported/BTL-3-Compact-macos-arm64/libexec/llama-server
|
| 66 |
+
8ee92d4b719fe2cb94cfe673beb55b51c2c4f6248b98449224462aa47346b068 runtimes/supported/BTL-3-Compact-macos-arm64/share/btl3/ollama_bridge.py
|
| 67 |
+
4c0c84652be9749c3aa521a9b75a2b4511e0d949f755b41eca960929717e1fc0 tools/install_consumer_bundle.py
|
docs/btl-3-compact-gguf-exporter.md
ADDED
|
@@ -0,0 +1,69 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# BTL-3 Compact GGUF exporter
|
| 2 |
+
|
| 3 |
+
## Scope
|
| 4 |
+
|
| 5 |
+
The exporter turns the frozen BTL-3 Compact package into the tensor contract
|
| 6 |
+
consumed by the custom Qwen3.6 llama.cpp backend. It never reconstructs a dense
|
| 7 |
+
decoder. Safetensors are mmap-backed and materialized one tensor at a time.
|
| 8 |
+
Every one of the 158 source files is size- and SHA-256-checked against the
|
| 9 |
+
frozen package manifest before planning or writing.
|
| 10 |
+
|
| 11 |
+
It exports:
|
| 12 |
+
|
| 13 |
+
- standard Qwen3.6 small-state tensors with official converter semantics;
|
| 14 |
+
- AVQ2 codes, affines, and exact Torch-seeded sign vectors;
|
| 15 |
+
- affine-INT4 codes, scales, and zero points;
|
| 16 |
+
- the two final INT4 demotions in place of their superseded BF16 islands;
|
| 17 |
+
- frozen BF16 islands;
|
| 18 |
+
- the rank-8 behavior LoRA;
|
| 19 |
+
- AVQ2 embedding and output tensors, rescued embedding rows, and head residual.
|
| 20 |
+
|
| 21 |
+
## Verified planning result
|
| 22 |
+
|
| 23 |
+
The full dry-run plans 2,416 tensors and 8,381,262,032 tensor bytes. The
|
| 24 |
+
170,510,920-byte reduction from the source package is expected: the package
|
| 25 |
+
retains two superseded BF16 island files for provenance, while the export emits
|
| 26 |
+
only their final INT4 demotions.
|
| 27 |
+
|
| 28 |
+
The exporter reports no unsupported representations and no native-runtime
|
| 29 |
+
contract gaps.
|
| 30 |
+
|
| 31 |
+
## Numerical conformance
|
| 32 |
+
|
| 33 |
+
Three representative layers were exported so every decoder representation was
|
| 34 |
+
materialized:
|
| 35 |
+
|
| 36 |
+
| Layer | Coverage | Tensors | Bytes | Payloads verified |
|
| 37 |
+
|---:|---|---:|---:|---:|
|
| 38 |
+
| 13 | AVQ2, BF16 island, final INT4 demotion | 28 | 322,295,104 | 28/28 |
|
| 39 |
+
| 40 | AVQ2, small state, behavior LoRA | 44 | 98,427,808 | 44/44 |
|
| 40 |
+
| 47 | AVQ2, standard affine INT4, behavior LoRA | 35 | 122,932,064 | 35/35 |
|
| 41 |
+
|
| 42 |
+
Every tensor passed:
|
| 43 |
+
|
| 44 |
+
- canonical-name equality;
|
| 45 |
+
- source-shape equality;
|
| 46 |
+
- exact GGUF storage-type equality;
|
| 47 |
+
- exact raw payload-byte equality after required Qwen transformations;
|
| 48 |
+
- exact regeneration of seeded signs.
|
| 49 |
+
|
| 50 |
+
The layer-40 artifact SHA-256 is
|
| 51 |
+
`e621b4cdb1e3f25cdde514427b14cd6d27bacbe72ad5850468311c693d4d79f7`.
|
| 52 |
+
|
| 53 |
+
This one-layer artifact is not a runnable model. A full export was deliberately
|
| 54 |
+
not started at this gate.
|
| 55 |
+
|
| 56 |
+
## Commands
|
| 57 |
+
|
| 58 |
+
```bash
|
| 59 |
+
.venv/bin/python tools/export_btl3_compact_gguf.py \
|
| 60 |
+
--source /path/to/BTL-3/compact \
|
| 61 |
+
--dry-run \
|
| 62 |
+
--report artifacts/btl3-compact-gguf-dry-run.json
|
| 63 |
+
|
| 64 |
+
.venv/bin/python tools/export_btl3_compact_gguf.py \
|
| 65 |
+
--source /path/to/BTL-3/compact \
|
| 66 |
+
--conformance-layer 40 \
|
| 67 |
+
--output artifacts/conformance/btl-3-compact-layer40.gguf \
|
| 68 |
+
--report artifacts/conformance/btl-3-compact-layer40.report.json
|
| 69 |
+
```
|
docs/launch-btl3-compact.md
ADDED
|
@@ -0,0 +1,153 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Launch BTL-3 Compact locally
|
| 2 |
+
|
| 3 |
+
This repository ships a relocatable macOS arm64 server bundle for the exact
|
| 4 |
+
BTL-3 Compact AVQ2 artifact:
|
| 5 |
+
|
| 6 |
+
- model: `BTL-3-Compact-AVQ2.gguf`
|
| 7 |
+
- bytes: `8,392,369,600`
|
| 8 |
+
- SHA-256:
|
| 9 |
+
`2ddf9527620a17a2a6739d184a7096c45712092e6589128792ec6254e94dc30c`
|
| 10 |
+
- runtime: llama.cpp build 9596, commit `9fcaed763`
|
| 11 |
+
|
| 12 |
+
The model is external to the 28 MB runtime bundle. The launcher finds the model
|
| 13 |
+
in `artifacts/release`, in the bundle's `model` directory, or at `BTL3_MODEL`.
|
| 14 |
+
|
| 15 |
+
## Build and verify the bundle
|
| 16 |
+
|
| 17 |
+
Run this on an Apple Silicon Mac with Homebrew OpenSSL 3 installed:
|
| 18 |
+
|
| 19 |
+
```bash
|
| 20 |
+
rm -rf artifacts/runtime/BTL-3-Compact-macos-arm64
|
| 21 |
+
.venv/bin/python tools/build_macos_arm64_bundle.py
|
| 22 |
+
```
|
| 23 |
+
|
| 24 |
+
The builder copies the native dependency closure, rewrites absolute install
|
| 25 |
+
names and rpaths, ad-hoc signs the result, includes dependency licenses, and
|
| 26 |
+
writes `bundle-manifest.json`.
|
| 27 |
+
|
| 28 |
+
## OpenAI-compatible API
|
| 29 |
+
|
| 30 |
+
Start the server:
|
| 31 |
+
|
| 32 |
+
```bash
|
| 33 |
+
BTL3_CTX_SIZE=4096 \
|
| 34 |
+
artifacts/runtime/BTL-3-Compact-macos-arm64/bin/btl3-server
|
| 35 |
+
```
|
| 36 |
+
|
| 37 |
+
Check it:
|
| 38 |
+
|
| 39 |
+
```bash
|
| 40 |
+
curl -s http://127.0.0.1:8080/health
|
| 41 |
+
curl -s http://127.0.0.1:8080/v1/models
|
| 42 |
+
```
|
| 43 |
+
|
| 44 |
+
Call chat completions with any OpenAI-compatible client:
|
| 45 |
+
|
| 46 |
+
```bash
|
| 47 |
+
curl http://127.0.0.1:8080/v1/chat/completions \
|
| 48 |
+
-H 'Content-Type: application/json' \
|
| 49 |
+
-d '{
|
| 50 |
+
"model": "BTL-3",
|
| 51 |
+
"messages": [{"role": "user", "content": "Write a retrying fetch helper."}],
|
| 52 |
+
"stream": true
|
| 53 |
+
}'
|
| 54 |
+
```
|
| 55 |
+
|
| 56 |
+
Useful configuration:
|
| 57 |
+
|
| 58 |
+
| Variable | Default | Meaning |
|
| 59 |
+
|---|---:|---|
|
| 60 |
+
| `BTL3_HOST` | `127.0.0.1` | Listen address |
|
| 61 |
+
| `BTL3_PORT` | `8080` | OpenAI API port |
|
| 62 |
+
| `BTL3_CTX_SIZE` | `32768` | Allocated context |
|
| 63 |
+
| `BTL3_PARALLEL` | `1` | Concurrent slots |
|
| 64 |
+
| `BTL3_GPU_LAYERS` | `99` | Layers requested on Metal |
|
| 65 |
+
| `BTL3_API_KEY` | unset | Optional local bearer key |
|
| 66 |
+
| `BTL3_MODEL_ALIASES` | `BTL-3` | Comma-separated transport aliases |
|
| 67 |
+
|
| 68 |
+
On a 16 GB Mac, begin at 2K–4K context. A larger context raises KV-cache and
|
| 69 |
+
working-memory requirements. The model's declared context length is not a
|
| 70 |
+
promise that a particular device can allocate it.
|
| 71 |
+
|
| 72 |
+
## LM Studio
|
| 73 |
+
|
| 74 |
+
LM Studio's official
|
| 75 |
+
[`openai-compat-endpoint`](https://www.lmstudio.ai/lmstudio/openai-compat-endpoint)
|
| 76 |
+
generator can target the local API. Its current model picker uses a fixed set
|
| 77 |
+
of model IDs, so expose one of those IDs as a **transport alias**:
|
| 78 |
+
|
| 79 |
+
```bash
|
| 80 |
+
BTL3_API_KEY=btl3-local \
|
| 81 |
+
BTL3_MODEL_ALIASES='BTL-3,gpt-4.1-2025-04-14' \
|
| 82 |
+
BTL3_CTX_SIZE=4096 \
|
| 83 |
+
artifacts/runtime/BTL-3-Compact-macos-arm64/bin/btl3-server
|
| 84 |
+
```
|
| 85 |
+
|
| 86 |
+
Then:
|
| 87 |
+
|
| 88 |
+
1. Install the official plugin from its LM Studio Hub page.
|
| 89 |
+
2. Set **Override Base URL** to `http://127.0.0.1:8080/v1`.
|
| 90 |
+
3. Set its API key to `btl3-local`.
|
| 91 |
+
4. Select `gpt-4.1-2025-04-14`.
|
| 92 |
+
|
| 93 |
+
That name is only protocol routing. The served model remains BTL-3; the
|
| 94 |
+
integration does not claim it is an OpenAI model.
|
| 95 |
+
|
| 96 |
+
## Ollama CLI
|
| 97 |
+
|
| 98 |
+
Stock Ollama does not load BTL-3's custom AVQ2 GGUF. Do not use
|
| 99 |
+
`ollama create`. The supplied bridge lets the unmodified Ollama CLI speak to
|
| 100 |
+
the native BTL-3 server through Ollama's local HTTP protocol.
|
| 101 |
+
|
| 102 |
+
In terminal one:
|
| 103 |
+
|
| 104 |
+
```bash
|
| 105 |
+
BTL3_CTX_SIZE=4096 \
|
| 106 |
+
artifacts/runtime/BTL-3-Compact-macos-arm64/bin/btl3-server
|
| 107 |
+
```
|
| 108 |
+
|
| 109 |
+
In terminal two:
|
| 110 |
+
|
| 111 |
+
```bash
|
| 112 |
+
BTL3_CTX_SIZE=4096 \
|
| 113 |
+
artifacts/runtime/BTL-3-Compact-macos-arm64/bin/btl3-ollama-bridge
|
| 114 |
+
```
|
| 115 |
+
|
| 116 |
+
Then use the installed Ollama CLI:
|
| 117 |
+
|
| 118 |
+
```bash
|
| 119 |
+
OLLAMA_HOST=http://127.0.0.1:11435 ollama list
|
| 120 |
+
OLLAMA_HOST=http://127.0.0.1:11435 ollama show btl3-compact:latest
|
| 121 |
+
OLLAMA_HOST=http://127.0.0.1:11435 ollama run btl3-compact:latest
|
| 122 |
+
```
|
| 123 |
+
|
| 124 |
+
The bridge implements `/api/chat`, `/api/generate`, `/api/tags`, `/api/show`,
|
| 125 |
+
`/api/ps`, and `/api/version`. It maps streaming to Ollama NDJSON and preserves
|
| 126 |
+
reasoning, tool calls, structured output, cancellation, token counts, and the
|
| 127 |
+
common sampling options. Unsupported Ollama management endpoints return 501
|
| 128 |
+
instead of pretending to work.
|
| 129 |
+
|
| 130 |
+
## Validation status
|
| 131 |
+
|
| 132 |
+
The packaged executable has passed native model load, graph reservation,
|
| 133 |
+
`/health`, and `/v1/models` checks on Apple M2. Protocol tests pass using the
|
| 134 |
+
real installed Ollama CLI and a deterministic OpenAI-compatible upstream.
|
| 135 |
+
|
| 136 |
+
The native Metal runtime accelerates AVQ2, affine INT4, the vocabulary head,
|
| 137 |
+
and rescued and ordinary embedding rows without reconstructing dense weights.
|
| 138 |
+
On a base Apple M2 with 16 GB unified memory, a clean full-model smoke measured
|
| 139 |
+
2.30 prompt tokens/second and 2.48 generated tokens/second at a 128-token
|
| 140 |
+
allocated context. Treat those as one-device smoke results, not universal
|
| 141 |
+
throughput claims. Context size, prompt length, thermal state, and Apple chip
|
| 142 |
+
generation will change performance.
|
| 143 |
+
|
| 144 |
+
The exact GGUF has passed full-model native CUDA execution on an RTX PRO 6000
|
| 145 |
+
Blackwell Server Edition. Three repetitions measured 84.70 prompt
|
| 146 |
+
tokens/second and 43.16 generated tokens/second with full GPU offload. This
|
| 147 |
+
does not establish RTX 4090, RTX 5090, DGX Spark, or Windows compatibility;
|
| 148 |
+
each target still requires its own packaged-runtime gate. Do not reuse the old
|
| 149 |
+
Python/Triton H100 number as a native-runtime claim.
|
| 150 |
+
|
| 151 |
+
The native server may display roughly 7.7B `n_params`; that value counts packed
|
| 152 |
+
stored elements, not the logical 27B architecture. Product metadata exposed by
|
| 153 |
+
the bridge reports the logical model class.
|
docs/launch-btl3-cuda.md
ADDED
|
@@ -0,0 +1,161 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# BTL-3 consumer NVIDIA build and packaging
|
| 2 |
+
|
| 3 |
+
This repository contains reproducible native CUDA build and package definitions
|
| 4 |
+
for three consumer targets:
|
| 5 |
+
|
| 6 |
+
| Target | Intended hardware | Requested CUDA architectures |
|
| 7 |
+
|---|---|---|
|
| 8 |
+
| `linux-x86_64` | RTX 4090 and RTX 5090 | `89-real;120-real` |
|
| 9 |
+
| `windows-x86_64` | RTX 4090 and RTX 5090 | `89-real;120-real` |
|
| 10 |
+
| `linux-arm64` | NVIDIA DGX Spark | `121-real` |
|
| 11 |
+
|
| 12 |
+
The definitions pin CUDA Toolkit 13.0.2. NVIDIA lists compute capability 8.9
|
| 13 |
+
for RTX 4090, 12.0 for RTX 5090, and 12.1 for DGX Spark. NVIDIA's DGX Spark
|
| 14 |
+
porting guide specifically recommends `CMAKE_CUDA_ARCHITECTURES="121-real"`.
|
| 15 |
+
The current llama.cpp fork deliberately rewrites requested plain Blackwell
|
| 16 |
+
architectures to architecture-specific code: `120-real` becomes `120a-real`
|
| 17 |
+
and `121-real` becomes `121a-real`. This enables Blackwell-specific
|
| 18 |
+
instructions and is not forward-compatible with later architectures.
|
| 19 |
+
|
| 20 |
+
Sources:
|
| 21 |
+
|
| 22 |
+
- <https://developer.nvidia.com/cuda/gpus>
|
| 23 |
+
- <https://docs.nvidia.com/dgx/dgx-spark-porting-guide/porting/compilation.html>
|
| 24 |
+
- <https://docs.nvidia.com/dgx/dgx-spark/system-overview.html>
|
| 25 |
+
- <https://docs.nvidia.com/dgx/dgx-spark/release-notes.html>
|
| 26 |
+
|
| 27 |
+
## Current support boundary
|
| 28 |
+
|
| 29 |
+
Native AVQ2, affine INT4, vocabulary projection, and embedding kernels have
|
| 30 |
+
passed exact-GGUF full-model execution on one Linux x86_64 RTX PRO 6000
|
| 31 |
+
Blackwell Server Edition (`sm_120a`). Three native benchmark repetitions
|
| 32 |
+
measured 84.70 prompt tokens/second and 43.16 generated tokens/second.
|
| 33 |
+
|
| 34 |
+
That measurement validates the implementation on that device. It does not
|
| 35 |
+
automatically validate a separately built archive, RTX 4090, RTX 5090,
|
| 36 |
+
Windows, or DGX Spark. The complete Linux arm64 runner cross-compiles under
|
| 37 |
+
CUDA 13.0.2 for DGX Spark `sm_121a`, but remains a preview until target-device
|
| 38 |
+
execution. Do not advertise unmeasured targets as supported.
|
| 39 |
+
|
| 40 |
+
## Linux x86_64 and DGX Spark builds
|
| 41 |
+
|
| 42 |
+
The Docker definition uses NVIDIA's multi-architecture
|
| 43 |
+
`nvidia/cuda:13.0.2-devel-ubuntu24.04` image. Its published manifest contains
|
| 44 |
+
both `linux/amd64` and `linux/arm64`. The Dockerfile pins manifest-list digest
|
| 45 |
+
`sha256:5dc1bca23d05bd37b011be68ec470c03b403a5da07ec3a86e41af9470e9d0cc6`
|
| 46 |
+
so the tag cannot silently change the toolchain.
|
| 47 |
+
|
| 48 |
+
Build an RTX Linux bundle:
|
| 49 |
+
|
| 50 |
+
```bash
|
| 51 |
+
packaging/cuda/build-linux.sh \
|
| 52 |
+
linux-x86_64 artifacts/runtime/BTL-3-Compact-linux-x86_64-cuda
|
| 53 |
+
```
|
| 54 |
+
|
| 55 |
+
Build natively on DGX Spark:
|
| 56 |
+
|
| 57 |
+
```bash
|
| 58 |
+
packaging/cuda/build-linux.sh \
|
| 59 |
+
linux-arm64 artifacts/runtime/BTL-3-Compact-linux-arm64-cuda
|
| 60 |
+
```
|
| 61 |
+
|
| 62 |
+
Cross-building arm64 through Docker Buildx is useful as a compile/package
|
| 63 |
+
check, but it does not replace execution on DGX Spark.
|
| 64 |
+
|
| 65 |
+
## Windows x64 build
|
| 66 |
+
|
| 67 |
+
Run this in PowerShell with Visual Studio C++ tools, CMake, Python, and CUDA
|
| 68 |
+
Toolkit 13.0.x installed:
|
| 69 |
+
|
| 70 |
+
```powershell
|
| 71 |
+
.\packaging\cuda\build-windows.ps1 `
|
| 72 |
+
-Output artifacts\runtime\BTL-3-Compact-windows-x86_64-cuda
|
| 73 |
+
```
|
| 74 |
+
|
| 75 |
+
The package contains `llama-server.exe`, `llama-cli.exe`, the produced llama
|
| 76 |
+
and GGML DLLs, and the required redistributable CUDA runtime DLLs. The NVIDIA
|
| 77 |
+
display driver remains a system prerequisite.
|
| 78 |
+
|
| 79 |
+
## Bundle contents
|
| 80 |
+
|
| 81 |
+
The model is deliberately external to every runtime bundle:
|
| 82 |
+
|
| 83 |
+
- `libexec/llama-server[.exe]`
|
| 84 |
+
- `libexec/llama-cli[.exe]`
|
| 85 |
+
- `lib/` runtime dependency closure
|
| 86 |
+
- `bin/btl3-server[.ps1]`
|
| 87 |
+
- `bundle-manifest.json`
|
| 88 |
+
- `model/` empty destination for the model
|
| 89 |
+
|
| 90 |
+
The manifest records checksums for packaged files and the expected external
|
| 91 |
+
model:
|
| 92 |
+
|
| 93 |
+
- filename: `BTL-3-Compact-AVQ2.gguf`
|
| 94 |
+
- bytes: `8,392,369,600`
|
| 95 |
+
- SHA-256:
|
| 96 |
+
`2ddf9527620a17a2a6739d184a7096c45712092e6589128792ec6254e94dc30c`
|
| 97 |
+
|
| 98 |
+
## Safe context defaults
|
| 99 |
+
|
| 100 |
+
The launcher detects the first NVIDIA GPU's total memory. DGX Spark falls back
|
| 101 |
+
to system memory because CPU and GPU share its 128 GB unified pool.
|
| 102 |
+
|
| 103 |
+
| Detected memory | Default context |
|
| 104 |
+
|---:|---:|
|
| 105 |
+
| below 20,000 MiB | 16,384 |
|
| 106 |
+
| 20,000–27,999 MiB | 32,768 |
|
| 107 |
+
| 28,000–47,999 MiB | 65,536 |
|
| 108 |
+
| 48,000–95,999 MiB | 98,304 |
|
| 109 |
+
| at least 96,000 MiB | 131,072 |
|
| 110 |
+
|
| 111 |
+
These defaults reserve memory for the 8.39 GB model, runtime workspace, and
|
| 112 |
+
other processes. Override them with `BTL3_CTX_SIZE`. Override faulty hardware
|
| 113 |
+
detection with `BTL3_GPU_MEMORY_MIB`. On DGX Spark, the launcher enables
|
| 114 |
+
`GGML_CUDA_ENABLE_UNIFIED_MEMORY=1`; lower context if the system is under
|
| 115 |
+
memory pressure.
|
| 116 |
+
|
| 117 |
+
Place the GGUF in the bundle's `model` directory or set `BTL3_MODEL`, then run:
|
| 118 |
+
|
| 119 |
+
```bash
|
| 120 |
+
BTL3_PRINT_COMMAND=1 ./bin/btl3-server
|
| 121 |
+
./bin/btl3-server
|
| 122 |
+
```
|
| 123 |
+
|
| 124 |
+
For a verified per-user installation used automatically by the LM Studio
|
| 125 |
+
plugin, install the downloaded runtime and GGUF in one command:
|
| 126 |
+
|
| 127 |
+
```bash
|
| 128 |
+
python3 tools/install_consumer_bundle.py \
|
| 129 |
+
--runtime artifacts/runtime/BTL-3-Compact-linux-x86_64-cuda \
|
| 130 |
+
--model artifacts/release/BTL-3-Compact-AVQ2.gguf
|
| 131 |
+
```
|
| 132 |
+
|
| 133 |
+
The installer checks every runtime hash plus the complete 8.39 GB model hash
|
| 134 |
+
before atomically installing. Its defaults are `~/.local/share/btl3` on Linux
|
| 135 |
+
and `%LOCALAPPDATA%\BTL3` on Windows.
|
| 136 |
+
|
| 137 |
+
On Windows:
|
| 138 |
+
|
| 139 |
+
```powershell
|
| 140 |
+
$env:BTL3_PRINT_COMMAND = "1"
|
| 141 |
+
.\bin\btl3-server.ps1
|
| 142 |
+
```
|
| 143 |
+
|
| 144 |
+
The print mode validates model discovery, memory detection, and chosen context
|
| 145 |
+
without starting the server.
|
| 146 |
+
|
| 147 |
+
## Required NVIDIA conformance
|
| 148 |
+
|
| 149 |
+
Before changing the support label:
|
| 150 |
+
|
| 151 |
+
1. Build both Linux architectures and Windows x64 from clean environments.
|
| 152 |
+
2. Run `test-btl3-avq-cuda`, `test-btl3-int4-cuda`, and
|
| 153 |
+
`test-btl3-vocab-cuda` on RTX 4090, RTX 5090, and DGX Spark.
|
| 154 |
+
3. Record numerical parity and packed-probe placement for all four custom
|
| 155 |
+
operation families.
|
| 156 |
+
4. Load the exact external GGUF and execute prompt plus decode.
|
| 157 |
+
5. Run transport, reasoning, tool-call, cancellation, and behavior gates.
|
| 158 |
+
6. Record throughput, peak memory, driver, CUDA runtime, and artifact hashes.
|
| 159 |
+
|
| 160 |
+
No throughput or compatibility number should be inferred from a successful
|
| 161 |
+
cross-compile.
|
docs/patched-ollama-and-lmstudio.md
ADDED
|
@@ -0,0 +1,234 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# BTL-3 consumer integrations
|
| 2 |
+
|
| 3 |
+
This document separates three products that must not be conflated:
|
| 4 |
+
|
| 5 |
+
| Surface | How BTL-3 executes | Current status |
|
| 6 |
+
|---|---|---|
|
| 7 |
+
| Stock Ollama | Stock runner | Does not execute AVQ2 |
|
| 8 |
+
| **BTL-3 Patched Ollama** | Ollama directly spawns BTL's native `llama-server` | Packaging scaffold complete; target packages pending |
|
| 9 |
+
| LM Studio | Official generator plugin calls BTL's native local endpoint | Plugin type-checks; macOS native endpoint verified |
|
| 10 |
+
|
| 11 |
+
No HTTP translation bridge sits between patched Ollama and the runner.
|
| 12 |
+
|
| 13 |
+
## Why the runner replacement is small
|
| 14 |
+
|
| 15 |
+
At official Ollama commit
|
| 16 |
+
[`573386c35eac76124ffce571f4b0fefa0a7fe13c`](https://github.com/ollama/ollama/commit/573386c35eac76124ffce571f4b0fefa0a7fe13c),
|
| 17 |
+
`llm.NewLlamaServer` delegates every GGUF model to a `llama-server`
|
| 18 |
+
subprocess. Ollama locates that executable under `lib/ollama`, starts it on an
|
| 19 |
+
ephemeral loopback port, and speaks its native `/health`, `/completion`,
|
| 20 |
+
`/v1/chat/completions`, tokenization, and embedding endpoints. The relevant
|
| 21 |
+
upstream files are
|
| 22 |
+
[`llm/server.go`](https://github.com/ollama/ollama/blob/573386c35eac76124ffce571f4b0fefa0a7fe13c/llm/server.go),
|
| 23 |
+
[`llm/llama_server.go`](https://github.com/ollama/ollama/blob/573386c35eac76124ffce571f4b0fefa0a7fe13c/llm/llama_server.go),
|
| 24 |
+
and
|
| 25 |
+
[`llm/llama_binary.go`](https://github.com/ollama/ollama/blob/573386c35eac76124ffce571f4b0fefa0a7fe13c/llm/llama_binary.go).
|
| 26 |
+
|
| 27 |
+
BTL's native runner already exposes that contract. The patched distribution
|
| 28 |
+
replaces Ollama's runtime payload, not its public API, scheduler, CLI, model
|
| 29 |
+
store, or UI.
|
| 30 |
+
|
| 31 |
+
## Build the CUDA runner payload
|
| 32 |
+
|
| 33 |
+
Build on the target machine so CMake selects the target GPU:
|
| 34 |
+
|
| 35 |
+
```bash
|
| 36 |
+
cmake -S native/llama.cpp -B native/llama.cpp/build-btl3-cuda \
|
| 37 |
+
-DCMAKE_BUILD_TYPE=Release \
|
| 38 |
+
-DCMAKE_CUDA_ARCHITECTURES=native \
|
| 39 |
+
-DBUILD_SHARED_LIBS=ON \
|
| 40 |
+
-DGGML_BACKEND_DL=ON \
|
| 41 |
+
-DGGML_CUDA=ON \
|
| 42 |
+
-DLLAMA_CURL=OFF
|
| 43 |
+
cmake --build native/llama.cpp/build-btl3-cuda \
|
| 44 |
+
--target llama-server --parallel
|
| 45 |
+
```
|
| 46 |
+
|
| 47 |
+
Stage the executable and its shared libraries. Use `cuda_v13` with CUDA 13 and
|
| 48 |
+
`cuda_v12` with CUDA 12:
|
| 49 |
+
|
| 50 |
+
```bash
|
| 51 |
+
payload=/tmp/btl3-runner
|
| 52 |
+
cuda_dir=cuda_v13
|
| 53 |
+
rm -rf "$payload"
|
| 54 |
+
mkdir -p "$payload/$cuda_dir"
|
| 55 |
+
cp native/llama.cpp/build-btl3-cuda/bin/llama-server "$payload/"
|
| 56 |
+
cp native/llama.cpp/build-btl3-cuda/bin/libggml*.so* "$payload/"
|
| 57 |
+
cp native/llama.cpp/build-btl3-cuda/bin/libllama*.so* "$payload/"
|
| 58 |
+
mv "$payload"/libggml-cuda.so* "$payload/$cuda_dir/"
|
| 59 |
+
"$payload/llama-server" --help
|
| 60 |
+
```
|
| 61 |
+
|
| 62 |
+
Do not package until `--help` exposes `--model`, `--host`, `--port`,
|
| 63 |
+
`--no-webui`, `--offline`, and `-np`. The packager enforces that contract.
|
| 64 |
+
|
| 65 |
+
### Hardware targets
|
| 66 |
+
|
| 67 |
+
- RTX 4090: build `linux-amd64-cuda12` or `cuda13` on that machine.
|
| 68 |
+
- RTX 5090: build on the 5090 so `native` includes its Blackwell target.
|
| 69 |
+
- Windows RTX 4090/5090: build `windows-amd64-cuda13` on Windows x64.
|
| 70 |
+
- DGX Spark: build `linux-arm64-cuda13` on Spark. Current DGX Spark software is
|
| 71 |
+
ARM64 and ships CUDA 13; do not reuse an x86_64 payload. See NVIDIA's
|
| 72 |
+
[hardware overview](https://docs.nvidia.com/dgx/dgx-spark/hardware.html) and
|
| 73 |
+
[current release notes](https://docs.nvidia.com/dgx/dgx-spark/release-notes.html).
|
| 74 |
+
|
| 75 |
+
## Build BTL-3 Patched Ollama
|
| 76 |
+
|
| 77 |
+
Build Ollama from the pinned source commit using its official Linux build, or
|
| 78 |
+
unpack a distribution built from that exact commit into `OLLAMA_DIST`. Then:
|
| 79 |
+
|
| 80 |
+
```bash
|
| 81 |
+
.venv/bin/python tools/build_patched_ollama.py \
|
| 82 |
+
--ollama-dist "$OLLAMA_DIST" \
|
| 83 |
+
--runner-payload /tmp/btl3-runner \
|
| 84 |
+
--platform linux-amd64-cuda13 \
|
| 85 |
+
--output dist/btl3-patched-ollama-linux-amd64-cuda13
|
| 86 |
+
```
|
| 87 |
+
|
| 88 |
+
The output contains:
|
| 89 |
+
|
| 90 |
+
```text
|
| 91 |
+
bin/ollama
|
| 92 |
+
bin/btl3-ollama
|
| 93 |
+
lib/ollama/llama-server
|
| 94 |
+
lib/ollama/cuda_v13/libggml-cuda.so
|
| 95 |
+
share/btl3/patched-ollama-manifest.json
|
| 96 |
+
```
|
| 97 |
+
|
| 98 |
+
`btl3-ollama --btl3-info` prints the patched label. All other arguments are
|
| 99 |
+
passed to the packaged Ollama binary.
|
| 100 |
+
|
| 101 |
+
Create the local model only with the patched distribution:
|
| 102 |
+
|
| 103 |
+
```bash
|
| 104 |
+
cp artifacts/release/BTL-3-Compact-AVQ2.gguf integrations/ollama/
|
| 105 |
+
dist/btl3-patched-ollama-linux-amd64-cuda13/bin/btl3-ollama serve
|
| 106 |
+
# in another terminal
|
| 107 |
+
dist/btl3-patched-ollama-linux-amd64-cuda13/bin/btl3-ollama \
|
| 108 |
+
create btl3-compact -f integrations/ollama/Modelfile
|
| 109 |
+
dist/btl3-patched-ollama-linux-amd64-cuda13/bin/btl3-ollama \
|
| 110 |
+
run btl3-compact
|
| 111 |
+
```
|
| 112 |
+
|
| 113 |
+
The GGUF stores BTL's packed representation in ordinary I8/F32/BF16 tensors
|
| 114 |
+
with custom names, so no new GGUF tensor-type enum is required. Actual model
|
| 115 |
+
creation and generation remain release gates on each CUDA target.
|
| 116 |
+
|
| 117 |
+
### Windows x64 CUDA 13
|
| 118 |
+
|
| 119 |
+
The pinned Ollama source has the same direct subprocess architecture on
|
| 120 |
+
Windows. Its
|
| 121 |
+
[`llama_binary.go`](https://github.com/ollama/ollama/blob/573386c35eac76124ffce571f4b0fefa0a7fe13c/llm/llama_binary.go)
|
| 122 |
+
adds `.exe` and searches beside a top-level `ollama.exe` at
|
| 123 |
+
`lib\ollama\llama-server.exe`; the pinned
|
| 124 |
+
[`build_windows.ps1`](https://github.com/ollama/ollama/blob/573386c35eac76124ffce571f4b0fefa0a7fe13c/scripts/build_windows.ps1)
|
| 125 |
+
installs CUDA payloads under that same directory.
|
| 126 |
+
|
| 127 |
+
Build the native runner from an x64 Visual Studio developer shell with CUDA 13:
|
| 128 |
+
|
| 129 |
+
```powershell
|
| 130 |
+
cmake -S native/llama.cpp -B native/llama.cpp/build-btl3-win-cuda13 `
|
| 131 |
+
-A x64 `
|
| 132 |
+
-DBUILD_SHARED_LIBS=ON `
|
| 133 |
+
-DGGML_BACKEND_DL=ON `
|
| 134 |
+
-DGGML_CUDA=ON `
|
| 135 |
+
-DLLAMA_CURL=OFF
|
| 136 |
+
cmake --build native/llama.cpp/build-btl3-win-cuda13 `
|
| 137 |
+
--config Release --target llama-server --parallel
|
| 138 |
+
|
| 139 |
+
$Bin = "native/llama.cpp/build-btl3-win-cuda13/bin/Release"
|
| 140 |
+
$Payload = "$env:TEMP/btl3-runner-win"
|
| 141 |
+
Remove-Item -Recurse -Force $Payload -ErrorAction SilentlyContinue
|
| 142 |
+
New-Item -ItemType Directory "$Payload/cuda_v13" | Out-Null
|
| 143 |
+
Copy-Item "$Bin/llama-server.exe" $Payload
|
| 144 |
+
Copy-Item "$Bin/*.dll" $Payload
|
| 145 |
+
Move-Item "$Payload/ggml-cuda.dll" "$Payload/cuda_v13/"
|
| 146 |
+
powershell -ExecutionPolicy Bypass -File tools/probe_windows_runner.ps1 `
|
| 147 |
+
-Runner "$Payload/llama-server.exe" `
|
| 148 |
+
-Output "$Payload/runner-cli-contract.json"
|
| 149 |
+
```
|
| 150 |
+
|
| 151 |
+
The target-side probe executes the exact runner and binds its required Ollama
|
| 152 |
+
CLI flags to its SHA-256. Cross-platform packaging refuses a missing, stale,
|
| 153 |
+
or wrong-platform contract. It also checks that `ollama.exe`,
|
| 154 |
+
`llama-server.exe`, and every staged DLL are PE x64.
|
| 155 |
+
|
| 156 |
+
Package an official pinned Windows x64 distribution:
|
| 157 |
+
|
| 158 |
+
```powershell
|
| 159 |
+
python tools/build_patched_ollama.py `
|
| 160 |
+
--ollama-dist "$env:TEMP/ollama-windows-amd64" `
|
| 161 |
+
--runner-payload "$env:TEMP/btl3-runner-win" `
|
| 162 |
+
--platform windows-amd64-cuda13 `
|
| 163 |
+
--output dist/btl3-patched-ollama-windows-amd64-cuda13
|
| 164 |
+
|
| 165 |
+
dist\btl3-patched-ollama-windows-amd64-cuda13\btl3-ollama.cmd --btl3-info
|
| 166 |
+
dist\btl3-patched-ollama-windows-amd64-cuda13\btl3-ollama.cmd serve
|
| 167 |
+
```
|
| 168 |
+
|
| 169 |
+
The Windows output uses top-level `ollama.exe` and `btl3-ollama.cmd`, with the
|
| 170 |
+
runner at `lib\ollama\llama-server.exe` and CUDA backend at
|
| 171 |
+
`lib\ollama\cuda_v13\ggml-cuda.dll`. This path is source-verified and
|
| 172 |
+
package-tested, but still requires generation conformance on real Windows
|
| 173 |
+
CUDA hardware before release.
|
| 174 |
+
|
| 175 |
+
## Required Ollama release gates
|
| 176 |
+
|
| 177 |
+
For each 4090, 5090, and DGX Spark artifact:
|
| 178 |
+
|
| 179 |
+
1. Verify runner and CUDA library architecture with `file`/`ldd` on Linux, or
|
| 180 |
+
`dumpbin /headers` and `/dependents` on Windows.
|
| 181 |
+
2. Run `btl3-ollama --btl3-info` and verify the manifest hashes.
|
| 182 |
+
3. Create the model in a temporary `OLLAMA_MODELS` directory.
|
| 183 |
+
4. Confirm Ollama itself spawns the packaged runner PID.
|
| 184 |
+
5. Exercise streaming chat, cancellation, reasoning, single and parallel tool
|
| 185 |
+
calls, JSON schema output, context reuse, and unload/reload.
|
| 186 |
+
6. Reverify the exact model SHA-256:
|
| 187 |
+
`2ddf9527620a17a2a6739d184a7096c45712092e6589128792ec6254e94dc30c`.
|
| 188 |
+
|
| 189 |
+
Until those checks pass on a real CUDA executable, the output is a packaging
|
| 190 |
+
candidate—not an Ollama-compatible release.
|
| 191 |
+
|
| 192 |
+
## LM Studio generator
|
| 193 |
+
|
| 194 |
+
LM Studio's official plugin API treats generators as replacement token
|
| 195 |
+
sources. The BTL plugin therefore calls the native runner's OpenAI-compatible
|
| 196 |
+
endpoint; it does not ask LM Studio's stock engine to interpret AVQ2. Official
|
| 197 |
+
generator and tool-call APIs are documented in LM Studio's
|
| 198 |
+
[generator introduction](https://lmstudio.ai/docs/typescript/plugins/generator)
|
| 199 |
+
and
|
| 200 |
+
[tool-calling guide](https://lmstudio.ai/docs/typescript/plugins/generator/tool-calling-generators).
|
| 201 |
+
|
| 202 |
+
Install the BTL native runner and model into the platform-default BTL-3 data
|
| 203 |
+
directory (or configure their paths in the plugin), then:
|
| 204 |
+
|
| 205 |
+
```bash
|
| 206 |
+
cd integrations/lmstudio/btl3-native
|
| 207 |
+
./run-local
|
| 208 |
+
```
|
| 209 |
+
|
| 210 |
+
The script installs the locked dependencies and runs the official `lms dev`
|
| 211 |
+
workflow. The model appears in LM Studio's model picker as the generator
|
| 212 |
+
plugin. Its default endpoint is `http://127.0.0.1:8080/v1`. When that endpoint
|
| 213 |
+
is offline, the Node plugin starts the installed native runner directly with
|
| 214 |
+
the configured model; no separately launched compatibility server is required.
|
| 215 |
+
|
| 216 |
+
The plugin preserves:
|
| 217 |
+
|
| 218 |
+
- streamed content;
|
| 219 |
+
- reasoning fragments through `reasoningType: "reasoning"`;
|
| 220 |
+
- cancellation through `ctl.abortSignal`;
|
| 221 |
+
- full chat history and tool results;
|
| 222 |
+
- parallel tool-call IDs, names, and JSON argument fragments;
|
| 223 |
+
- explicit propagation of cancellation, connection, and API failures.
|
| 224 |
+
|
| 225 |
+
Content and reasoning stream live. LM Studio's generator event lifecycle
|
| 226 |
+
represents one active tool call at a time and its argument-fragment callback
|
| 227 |
+
does not include a call ID. To preserve BTL-3's parallel calls correctly, the
|
| 228 |
+
plugin buffers tool fragments by upstream call index and emits each complete
|
| 229 |
+
lifecycle sequentially when the server stream ends. It does not misattribute
|
| 230 |
+
interleaved arguments merely to make the tool UI look live.
|
| 231 |
+
|
| 232 |
+
The included `model.yaml` is catalog metadata, not a claim that the stock LM
|
| 233 |
+
Studio GGUF engine supports AVQ2. It uses the explicit compatibility label
|
| 234 |
+
`btl3-avq2-native`.
|
evidence/BTL-3-Compact-AVQ2.report.json
ADDED
|
@@ -0,0 +1,45 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"schema_version": 1,
|
| 3 |
+
"model": "BTL-3 Compact",
|
| 4 |
+
"mode": "full",
|
| 5 |
+
"source_manifest_sha256": "a2b763323eed76d8f78fe5cbdf5a2349323b2c3d87dddc037714569946961116",
|
| 6 |
+
"source_weight_bytes": 8551772952,
|
| 7 |
+
"tensor_count": 2416,
|
| 8 |
+
"projected_tensor_bytes": 8381262032,
|
| 9 |
+
"components": {
|
| 10 |
+
"small-state": {
|
| 11 |
+
"tensors": 449,
|
| 12 |
+
"bytes": 57767936
|
| 13 |
+
},
|
| 14 |
+
"avq2": {
|
| 15 |
+
"tensors": 1615,
|
| 16 |
+
"bytes": 5498130320
|
| 17 |
+
},
|
| 18 |
+
"affine-int4": {
|
| 19 |
+
"tensors": 189,
|
| 20 |
+
"bytes": 875438080
|
| 21 |
+
},
|
| 22 |
+
"bf16-island": {
|
| 23 |
+
"tensors": 12,
|
| 24 |
+
"bytes": 1205862400
|
| 25 |
+
},
|
| 26 |
+
"affine-int4-demotion": {
|
| 27 |
+
"tensors": 6,
|
| 28 |
+
"bytes": 43909120
|
| 29 |
+
},
|
| 30 |
+
"behavior-lora": {
|
| 31 |
+
"tensors": 132,
|
| 32 |
+
"bytes": 32440320
|
| 33 |
+
},
|
| 34 |
+
"vocabulary": {
|
| 35 |
+
"tensors": 13,
|
| 36 |
+
"bytes": 667713856
|
| 37 |
+
}
|
| 38 |
+
},
|
| 39 |
+
"exporter_unsupported": [],
|
| 40 |
+
"native_runtime_gaps": [],
|
| 41 |
+
"output_bytes": 8392369600,
|
| 42 |
+
"output_sha256": "2ddf9527620a17a2a6739d184a7096c45712092e6589128792ec6254e94dc30c",
|
| 43 |
+
"verified_tensor_count": 2416,
|
| 44 |
+
"verified_payload_count": 2416
|
| 45 |
+
}
|
evidence/compact-validation.md
ADDED
|
@@ -0,0 +1,122 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# BTL-3 Compact native release validation
|
| 2 |
+
|
| 3 |
+
Date: 2026-07-19
|
| 4 |
+
|
| 5 |
+
## Candidate
|
| 6 |
+
|
| 7 |
+
- Public model: **BTL-3 Compact**.
|
| 8 |
+
- Model lineage: Qwen3.6-27B plus BTL RL0013 and the frozen behavior repair.
|
| 9 |
+
- Scope: text-only coding, tool use and agent behavior.
|
| 10 |
+
- Representation: full 64-layer AVQ2/UniSVQ decoder, two measured INT4
|
| 11 |
+
demotions, selected BF16 islands, packed vocabulary matrices, rank-32 head
|
| 12 |
+
correction and a small behavior adapter.
|
| 13 |
+
- Source payload bytes before GGUF packing: 8,572,070,080.
|
| 14 |
+
- Source weight bytes: 8,551,772,952.
|
| 15 |
+
- Portable GGUF bytes: 8,392,369,600.
|
| 16 |
+
- Portable GGUF SHA-256:
|
| 17 |
+
`2ddf9527620a17a2a6739d184a7096c45712092e6589128792ec6254e94dc30c`.
|
| 18 |
+
|
| 19 |
+
The package is complete enough to instantiate and generate without downloading
|
| 20 |
+
or loading the BF16 Qwen checkpoint.
|
| 21 |
+
|
| 22 |
+
## Native runtime proof
|
| 23 |
+
|
| 24 |
+
The standalone loader meta-initializes the Qwen3.6 text architecture and
|
| 25 |
+
installs only the package's small state, packed decoder tensors, packed
|
| 26 |
+
embedding, packed head, rank-32 output correction and behavior LoRA.
|
| 27 |
+
|
| 28 |
+
The H100 smoke test observed:
|
| 29 |
+
|
| 30 |
+
- no surviving dense compatible decoder matrices;
|
| 31 |
+
- exact AVQ2 CUDA-kernel parity with the unpacked reference;
|
| 32 |
+
- INT4 maximum absolute kernel error of `3.0517578125e-05`;
|
| 33 |
+
- standalone model peak CUDA allocation of 8,552,500,736 bytes;
|
| 34 |
+
- successful autoregressive generation.
|
| 35 |
+
|
| 36 |
+
The source payload was subsequently exported into the portable GGUF without
|
| 37 |
+
reconstructing dense weights. The exporter byte-verified all 2,416 payloads
|
| 38 |
+
and reported no unsupported tensors or native runtime gaps. The exact GGUF
|
| 39 |
+
then passed native llama.cpp generation on Apple Metal and NVIDIA CUDA.
|
| 40 |
+
MLX, WebGPU, phone execution, and stock-engine compatibility remain unverified.
|
| 41 |
+
|
| 42 |
+
## Fresh sealed gate
|
| 43 |
+
|
| 44 |
+
Benchmark ID: `btl-fresh-tool-gate-2026-07-19-v1`
|
| 45 |
+
|
| 46 |
+
Cases SHA-256:
|
| 47 |
+
`d656a7862e16e64ed3a359ba1de10f7eafefad77f6cf5a8264d60287e1890a45`
|
| 48 |
+
|
| 49 |
+
The gate was authored after compression and behavior-repair choices were
|
| 50 |
+
frozen. It contains 100 scored turns:
|
| 51 |
+
|
| 52 |
+
- 20 single calls;
|
| 53 |
+
- 20 parallel calls;
|
| 54 |
+
- 20 sequential calls;
|
| 55 |
+
- 20 parallel-multiple calls;
|
| 56 |
+
- 20 abstention decisions.
|
| 57 |
+
|
| 58 |
+
All tool families are first-party and use a new `lumenharbor_*` namespace.
|
| 59 |
+
Mechanical QA found no schema errors, duplicate IDs, parse errors or exact tool
|
| 60 |
+
name overlap with repository training/evaluation data.
|
| 61 |
+
|
| 62 |
+
This is a private synthetic contract-retention gate. It is not a public coding
|
| 63 |
+
benchmark and must not be presented as a frontier benchmark score.
|
| 64 |
+
|
| 65 |
+
## Full-precision teacher
|
| 66 |
+
|
| 67 |
+
The frozen RL0013 teacher scored:
|
| 68 |
+
|
| 69 |
+
| Category | Correct | Total |
|
| 70 |
+
|---|---:|---:|
|
| 71 |
+
| Single | 20 | 20 |
|
| 72 |
+
| Parallel | 20 | 20 |
|
| 73 |
+
| Sequential | 20 | 20 |
|
| 74 |
+
| Parallel-multiple | 10 | 20 |
|
| 75 |
+
| Abstention | 20 | 20 |
|
| 76 |
+
| Overall | 90 | 100 |
|
| 77 |
+
|
| 78 |
+
The release metric is conditional retention on these 90 teacher-correct turns,
|
| 79 |
+
reported separately for every category and overall. Absolute student accuracy
|
| 80 |
+
is also retained in the raw result.
|
| 81 |
+
|
| 82 |
+
## Standalone result
|
| 83 |
+
|
| 84 |
+
The standalone package scored:
|
| 85 |
+
|
| 86 |
+
| Category | Student correct | Total | Teacher-correct retained | Retention |
|
| 87 |
+
|---|---:|---:|---:|---:|
|
| 88 |
+
| Single | 20 | 20 | 20 / 20 | 100% |
|
| 89 |
+
| Parallel | 20 | 20 | 20 / 20 | 100% |
|
| 90 |
+
| Sequential | 20 | 20 | 20 / 20 | 100% |
|
| 91 |
+
| Parallel-multiple | 3 | 20 | 3 / 10 | 30% |
|
| 92 |
+
| Abstention | 20 | 20 | 20 / 20 | 100% |
|
| 93 |
+
| **Overall** | **83** | **100** | **83 / 90** | **92.2%** |
|
| 94 |
+
|
| 95 |
+
All generations stopped. The measured malformed rate was 7%, entirely within
|
| 96 |
+
the difficult parallel-multiple family.
|
| 97 |
+
|
| 98 |
+
The seven teacher-correct/student-wrong parallel-multiple cases were not parser
|
| 99 |
+
false negatives. The package emitted a fluent abstention instead of making the
|
| 100 |
+
three requested independent calls. This is a real over-abstention failure under
|
| 101 |
+
novel multi-call schemas.
|
| 102 |
+
|
| 103 |
+
### Decision
|
| 104 |
+
|
| 105 |
+
The package passes a 90% **overall** conditional-retention rule. It fails a 90%
|
| 106 |
+
**per-category** rule because parallel-multiple retention is 30%.
|
| 107 |
+
|
| 108 |
+
Do not alter compression based on this sealed result. Any behavior repair aimed
|
| 109 |
+
at these cases creates a new candidate and requires a newly authored, untouched
|
| 110 |
+
release gate.
|
| 111 |
+
|
| 112 |
+
## Release boundary
|
| 113 |
+
|
| 114 |
+
This result validates that the exact text-only source payload is physically
|
| 115 |
+
standalone and retains more than 90% overall on the fresh CUDA gate. The GGUF
|
| 116 |
+
export preserved those payload bytes exactly. It does not support a claim of
|
| 117 |
+
uniformly preserved behavior or phone deployment.
|
| 118 |
+
|
| 119 |
+
The release includes a distributable macOS native runtime and exact-artifact
|
| 120 |
+
throughput measurements on Apple M2 and RTX PRO 6000. Other GPU packages,
|
| 121 |
+
stock Ollama/LM Studio execution, mobile runtimes, and a public compact-specific
|
| 122 |
+
coding benchmark remain separate gates.
|
evidence/native-load-report.json
ADDED
|
@@ -0,0 +1,23 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"schema_version": 1,
|
| 3 |
+
"model": "BTL-3 Compact",
|
| 4 |
+
"artifact": "BTL-3-Compact-AVQ2.gguf",
|
| 5 |
+
"artifact_bytes": 8392369600,
|
| 6 |
+
"artifact_sha256": "2ddf9527620a17a2a6739d184a7096c45712092e6589128792ec6254e94dc30c",
|
| 7 |
+
"tensor_count": 2416,
|
| 8 |
+
"verified_payload_count": 2416,
|
| 9 |
+
"runner_commit": "9fcaed763",
|
| 10 |
+
"platform": "macOS arm64",
|
| 11 |
+
"device": "Apple M2",
|
| 12 |
+
"context_tokens": 128,
|
| 13 |
+
"cpu_only": false,
|
| 14 |
+
"backend": "Apple Metal",
|
| 15 |
+
"model_load": "pass",
|
| 16 |
+
"graph_reservation": "pass",
|
| 17 |
+
"generation": "pass",
|
| 18 |
+
"prompt_tokens": 15,
|
| 19 |
+
"prompt_tokens_per_second": 2.297,
|
| 20 |
+
"generated_tokens": 4,
|
| 21 |
+
"generated_tokens_per_second": 2.483,
|
| 22 |
+
"generation_note": "Single Apple M2 full-model smoke; not a universal throughput claim"
|
| 23 |
+
}
|
evidence/rtx-pro-6000-speed-2026-07-19.md
ADDED
|
@@ -0,0 +1,59 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# BTL-3 Compact — RTX PRO 6000 speed
|
| 2 |
+
|
| 3 |
+
Measured on 2026-07-19 using the native patched `llama.cpp` CUDA runtime and
|
| 4 |
+
the frozen `BTL-3-Compact-AVQ2.gguf` release artifact.
|
| 5 |
+
|
| 6 |
+
## Artifact
|
| 7 |
+
|
| 8 |
+
- File bytes: `8,392,369,600`
|
| 9 |
+
- GPU offload: all layers (`-ngl 99`)
|
| 10 |
+
- KV types: FP16
|
| 11 |
+
- CUDA architecture: `sm_120a`
|
| 12 |
+
- GPU: NVIDIA RTX PRO 6000 Blackwell Server Edition
|
| 13 |
+
- GPU memory: 97,887 MiB reported by `nvidia-smi`
|
| 14 |
+
|
| 15 |
+
## Command
|
| 16 |
+
|
| 17 |
+
```text
|
| 18 |
+
llama-bench \
|
| 19 |
+
-m /vol/release/compact/BTL-3-Compact-AVQ2.gguf \
|
| 20 |
+
-ngl 99 \
|
| 21 |
+
-p 512 \
|
| 22 |
+
-n 128 \
|
| 23 |
+
-r 3 \
|
| 24 |
+
-o json
|
| 25 |
+
```
|
| 26 |
+
|
| 27 |
+
## Results
|
| 28 |
+
|
| 29 |
+
| Workload | Throughput | Standard deviation |
|
| 30 |
+
|---|---:|---:|
|
| 31 |
+
| 512-token prompt processing | 84.70 tok/s | 0.37 tok/s |
|
| 32 |
+
| 128-token generation | 43.16 tok/s | 0.29 tok/s |
|
| 33 |
+
|
| 34 |
+
Generation samples: `42.83`, `43.35`, and `43.30` tok/s.
|
| 35 |
+
|
| 36 |
+
Prompt-processing samples: `84.28`, `84.91`, and `84.91` tok/s.
|
| 37 |
+
|
| 38 |
+
The measured benchmark body, including model load, completed in 41.03 seconds.
|
| 39 |
+
|
| 40 |
+
## Consumer-GPU projections
|
| 41 |
+
|
| 42 |
+
These are projections, not measurements. Single-stream decode is scaled
|
| 43 |
+
primarily from memory bandwidth, then widened to account for architecture,
|
| 44 |
+
clock, power, kernel occupancy, and driver differences.
|
| 45 |
+
|
| 46 |
+
| GPU | Estimated generation | Estimated prompt processing | Fit |
|
| 47 |
+
|---|---:|---:|---|
|
| 48 |
+
| RTX 5090 32GB | 39–44 tok/s | 75–88 tok/s | Yes |
|
| 49 |
+
| RTX 4090 24GB | 22–28 tok/s | 45–60 tok/s | Yes |
|
| 50 |
+
| RTX 4050 Laptop 6GB | No credible full-GPU estimate | No credible full-GPU estimate | No |
|
| 51 |
+
|
| 52 |
+
The RTX PRO 6000 and RTX 5090 both advertise 1,792 GB/s memory bandwidth and
|
| 53 |
+
share the Blackwell generation, so near-parity is the reasonable batch-one
|
| 54 |
+
decode hypothesis. The RTX 4090 advertises 1,008 GB/s, or 56.25% of that
|
| 55 |
+
bandwidth. A literal RTX 4050 Laptop has only 6GB of VRAM, below the 8.39GB
|
| 56 |
+
weight file before KV cache and runtime allocations; it therefore requires
|
| 57 |
+
CPU offload and should be described as a low-single-digit, system-dependent
|
| 58 |
+
fallback rather than a supported full-GPU target.
|
| 59 |
+
|
integrations/btl3-native/README.md
ADDED
|
@@ -0,0 +1,29 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# BTL-3 Native for LM Studio
|
| 2 |
+
|
| 3 |
+
This generator exposes the native BTL-3 OpenAI-compatible server inside LM
|
| 4 |
+
Studio. It does not ask LM Studio's stock GGUF engine to load AVQ2.
|
| 5 |
+
|
| 6 |
+
## Local development install
|
| 7 |
+
|
| 8 |
+
1. Install the BTL-3 runtime and model in its platform-default location, or set
|
| 9 |
+
`BTL3_RUNNER_PATH` and `BTL3_MODEL_PATH`.
|
| 10 |
+
2. Install the LM Studio CLI with `npx lmstudio install-cli` if needed.
|
| 11 |
+
3. In this directory run `npm ci`, then `lms dev`.
|
| 12 |
+
4. Select **badtheorylabs/btl3-native** in LM Studio's model picker. The plugin
|
| 13 |
+
starts the native runner automatically when the local endpoint is offline.
|
| 14 |
+
|
| 15 |
+
The default endpoint is `http://127.0.0.1:8080/v1`. Change it in the plugin's
|
| 16 |
+
global settings when the native runner is elsewhere. Automatic startup can be
|
| 17 |
+
disabled when an independently managed server is preferred. The generator preserves
|
| 18 |
+
streamed content, reasoning fragments, cancellation, and parallel tool calls.
|
| 19 |
+
LM Studio's generator lifecycle has no per-fragment call ID, so parallel
|
| 20 |
+
tool-call fragments are buffered by call index and emitted sequentially after
|
| 21 |
+
the upstream stream finishes. This preserves the calls without misattributing
|
| 22 |
+
interleaved JSON fragments.
|
| 23 |
+
|
| 24 |
+
`model.yaml` is catalog metadata for the native-generator product. Its custom
|
| 25 |
+
`btl3-avq2-native` compatibility type deliberately avoids claiming that LM
|
| 26 |
+
Studio's stock GGUF engine can execute this model.
|
| 27 |
+
|
| 28 |
+
This plugin has not been published. `lms push` is intentionally outside this
|
| 29 |
+
repository's local validation workflow.
|
integrations/btl3-native/manifest.json
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"type": "plugin",
|
| 3 |
+
"runner": "node",
|
| 4 |
+
"owner": "badtheorylabs",
|
| 5 |
+
"name": "btl3-native",
|
| 6 |
+
"revision": 1
|
| 7 |
+
}
|
integrations/btl3-native/model.yaml
ADDED
|
@@ -0,0 +1,17 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Catalog metadata only. The btl3-native generator owns execution.
|
| 2 |
+
model: badtheorylabs/btl-3-compact
|
| 3 |
+
base: badtheorylabs/btl-3-compact-avq2-native
|
| 4 |
+
metadataOverrides:
|
| 5 |
+
domain: llm
|
| 6 |
+
architectures:
|
| 7 |
+
- qwen3.5
|
| 8 |
+
compatibilityTypes:
|
| 9 |
+
- btl3-avq2-native
|
| 10 |
+
paramsStrings:
|
| 11 |
+
- 27B
|
| 12 |
+
minMemoryUsageBytes: 10000000000
|
| 13 |
+
contextLengths:
|
| 14 |
+
- 262144
|
| 15 |
+
vision: false
|
| 16 |
+
reasoning: true
|
| 17 |
+
trainedForToolUse: true
|
integrations/btl3-native/package-lock.json
ADDED
|
@@ -0,0 +1,221 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"name": "btl3-native-lmstudio",
|
| 3 |
+
"version": "0.1.0",
|
| 4 |
+
"lockfileVersion": 3,
|
| 5 |
+
"requires": true,
|
| 6 |
+
"packages": {
|
| 7 |
+
"": {
|
| 8 |
+
"name": "btl3-native-lmstudio",
|
| 9 |
+
"version": "0.1.0",
|
| 10 |
+
"dependencies": {
|
| 11 |
+
"@lmstudio/sdk": "1.5.0",
|
| 12 |
+
"openai": "6.48.0"
|
| 13 |
+
},
|
| 14 |
+
"devDependencies": {
|
| 15 |
+
"@types/node": "24.0.0",
|
| 16 |
+
"typescript": "5.9.3"
|
| 17 |
+
}
|
| 18 |
+
},
|
| 19 |
+
"node_modules/@lmstudio/lms-isomorphic": {
|
| 20 |
+
"version": "0.4.6",
|
| 21 |
+
"resolved": "https://registry.npmjs.org/@lmstudio/lms-isomorphic/-/lms-isomorphic-0.4.6.tgz",
|
| 22 |
+
"integrity": "sha512-v0LIjXKnDe3Ff3XZO5eQjlVxTjleUHXaom14MV7QU9bvwaoo3l5p71+xJ3mmSaqZq370CQ6pTKCn1Bb7Jf+VwQ==",
|
| 23 |
+
"license": "Apache-2.0",
|
| 24 |
+
"dependencies": {
|
| 25 |
+
"ws": "^8.16.0"
|
| 26 |
+
}
|
| 27 |
+
},
|
| 28 |
+
"node_modules/@lmstudio/sdk": {
|
| 29 |
+
"version": "1.5.0",
|
| 30 |
+
"resolved": "https://registry.npmjs.org/@lmstudio/sdk/-/sdk-1.5.0.tgz",
|
| 31 |
+
"integrity": "sha512-fdY12x4hb14PEjYijh7YeCqT1ZDY5Ok6VR4l4+E/dI+F6NW8oB+P83Sxed5vqE4XgTzbgyPuSR2ZbMNxxF+6jA==",
|
| 32 |
+
"license": "Apache-2.0",
|
| 33 |
+
"dependencies": {
|
| 34 |
+
"@lmstudio/lms-isomorphic": "^0.4.6",
|
| 35 |
+
"chalk": "^4.1.2",
|
| 36 |
+
"jsonschema": "^1.5.0",
|
| 37 |
+
"zod": "^3.22.4",
|
| 38 |
+
"zod-to-json-schema": "^3.22.5"
|
| 39 |
+
}
|
| 40 |
+
},
|
| 41 |
+
"node_modules/@types/node": {
|
| 42 |
+
"version": "24.0.0",
|
| 43 |
+
"resolved": "https://registry.npmjs.org/@types/node/-/node-24.0.0.tgz",
|
| 44 |
+
"integrity": "sha512-yZQa2zm87aRVcqDyH5+4Hv9KYgSdgwX1rFnGvpbzMaC7YAljmhBET93TPiTd3ObwTL+gSpIzPKg5BqVxdCvxKg==",
|
| 45 |
+
"dev": true,
|
| 46 |
+
"license": "MIT",
|
| 47 |
+
"dependencies": {
|
| 48 |
+
"undici-types": "~7.8.0"
|
| 49 |
+
}
|
| 50 |
+
},
|
| 51 |
+
"node_modules/ansi-styles": {
|
| 52 |
+
"version": "4.3.0",
|
| 53 |
+
"resolved": "https://registry.npmjs.org/ansi-styles/-/ansi-styles-4.3.0.tgz",
|
| 54 |
+
"integrity": "sha512-zbB9rCJAT1rbjiVDb2hqKFHNYLxgtk8NURxZ3IZwD3F6NtxbXZQCnnSi1Lkx+IDohdPlFp222wVALIheZJQSEg==",
|
| 55 |
+
"license": "MIT",
|
| 56 |
+
"dependencies": {
|
| 57 |
+
"color-convert": "^2.0.1"
|
| 58 |
+
},
|
| 59 |
+
"engines": {
|
| 60 |
+
"node": ">=8"
|
| 61 |
+
},
|
| 62 |
+
"funding": {
|
| 63 |
+
"url": "https://github.com/chalk/ansi-styles?sponsor=1"
|
| 64 |
+
}
|
| 65 |
+
},
|
| 66 |
+
"node_modules/chalk": {
|
| 67 |
+
"version": "4.1.2",
|
| 68 |
+
"resolved": "https://registry.npmjs.org/chalk/-/chalk-4.1.2.tgz",
|
| 69 |
+
"integrity": "sha512-oKnbhFyRIXpUuez8iBMmyEa4nbj4IOQyuhc/wy9kY7/WVPcwIO9VA668Pu8RkO7+0G76SLROeyw9CpQ061i4mA==",
|
| 70 |
+
"license": "MIT",
|
| 71 |
+
"dependencies": {
|
| 72 |
+
"ansi-styles": "^4.1.0",
|
| 73 |
+
"supports-color": "^7.1.0"
|
| 74 |
+
},
|
| 75 |
+
"engines": {
|
| 76 |
+
"node": ">=10"
|
| 77 |
+
},
|
| 78 |
+
"funding": {
|
| 79 |
+
"url": "https://github.com/chalk/chalk?sponsor=1"
|
| 80 |
+
}
|
| 81 |
+
},
|
| 82 |
+
"node_modules/color-convert": {
|
| 83 |
+
"version": "2.0.1",
|
| 84 |
+
"resolved": "https://registry.npmjs.org/color-convert/-/color-convert-2.0.1.tgz",
|
| 85 |
+
"integrity": "sha512-RRECPsj7iu/xb5oKYcsFHSppFNnsj/52OVTRKb4zP5onXwVF3zVmmToNcOfGC+CRDpfK/U584fMg38ZHCaElKQ==",
|
| 86 |
+
"license": "MIT",
|
| 87 |
+
"dependencies": {
|
| 88 |
+
"color-name": "~1.1.4"
|
| 89 |
+
},
|
| 90 |
+
"engines": {
|
| 91 |
+
"node": ">=7.0.0"
|
| 92 |
+
}
|
| 93 |
+
},
|
| 94 |
+
"node_modules/color-name": {
|
| 95 |
+
"version": "1.1.4",
|
| 96 |
+
"resolved": "https://registry.npmjs.org/color-name/-/color-name-1.1.4.tgz",
|
| 97 |
+
"integrity": "sha512-dOy+3AuW3a2wNbZHIuMZpTcgjGuLU/uBL/ubcZF9OXbDo8ff4O8yVp5Bf0efS8uEoYo5q4Fx7dY9OgQGXgAsQA==",
|
| 98 |
+
"license": "MIT"
|
| 99 |
+
},
|
| 100 |
+
"node_modules/has-flag": {
|
| 101 |
+
"version": "4.0.0",
|
| 102 |
+
"resolved": "https://registry.npmjs.org/has-flag/-/has-flag-4.0.0.tgz",
|
| 103 |
+
"integrity": "sha512-EykJT/Q1KjTWctppgIAgfSO0tKVuZUjhgMr17kqTumMl6Afv3EISleU7qZUzoXDFTAHTDC4NOoG/ZxU3EvlMPQ==",
|
| 104 |
+
"license": "MIT",
|
| 105 |
+
"engines": {
|
| 106 |
+
"node": ">=8"
|
| 107 |
+
}
|
| 108 |
+
},
|
| 109 |
+
"node_modules/jsonschema": {
|
| 110 |
+
"version": "1.5.0",
|
| 111 |
+
"resolved": "https://registry.npmjs.org/jsonschema/-/jsonschema-1.5.0.tgz",
|
| 112 |
+
"integrity": "sha512-K+A9hhqbn0f3pJX17Q/7H6yQfD/5OXgdrR5UE12gMXCiN9D5Xq2o5mddV2QEcX/bjla99ASsAAQUyMCCRWAEhw==",
|
| 113 |
+
"license": "MIT",
|
| 114 |
+
"engines": {
|
| 115 |
+
"node": "*"
|
| 116 |
+
}
|
| 117 |
+
},
|
| 118 |
+
"node_modules/openai": {
|
| 119 |
+
"version": "6.48.0",
|
| 120 |
+
"resolved": "https://registry.npmjs.org/openai/-/openai-6.48.0.tgz",
|
| 121 |
+
"integrity": "sha512-KhVp+FyV50QrXNextvL9hIU5l6ox5HYuKQjGVk7lIqprgJol90+dQXWONV6S1lRWsKA1bXjrow8RsUT14M1hNA==",
|
| 122 |
+
"license": "Apache-2.0",
|
| 123 |
+
"peerDependencies": {
|
| 124 |
+
"@aws-sdk/credential-provider-node": ">=3.972.0 <4",
|
| 125 |
+
"@smithy/hash-node": ">=4.3.0 <5",
|
| 126 |
+
"@smithy/signature-v4": ">=5.4.0 <6",
|
| 127 |
+
"ws": "^8.18.0",
|
| 128 |
+
"zod": "^3.25 || ^4.0"
|
| 129 |
+
},
|
| 130 |
+
"peerDependenciesMeta": {
|
| 131 |
+
"@aws-sdk/credential-provider-node": {
|
| 132 |
+
"optional": true
|
| 133 |
+
},
|
| 134 |
+
"@smithy/hash-node": {
|
| 135 |
+
"optional": true
|
| 136 |
+
},
|
| 137 |
+
"@smithy/signature-v4": {
|
| 138 |
+
"optional": true
|
| 139 |
+
},
|
| 140 |
+
"ws": {
|
| 141 |
+
"optional": true
|
| 142 |
+
},
|
| 143 |
+
"zod": {
|
| 144 |
+
"optional": true
|
| 145 |
+
}
|
| 146 |
+
}
|
| 147 |
+
},
|
| 148 |
+
"node_modules/supports-color": {
|
| 149 |
+
"version": "7.2.0",
|
| 150 |
+
"resolved": "https://registry.npmjs.org/supports-color/-/supports-color-7.2.0.tgz",
|
| 151 |
+
"integrity": "sha512-qpCAvRl9stuOHveKsn7HncJRvv501qIacKzQlO/+Lwxc9+0q2wLyv4Dfvt80/DPn2pqOBsJdDiogXGR9+OvwRw==",
|
| 152 |
+
"license": "MIT",
|
| 153 |
+
"dependencies": {
|
| 154 |
+
"has-flag": "^4.0.0"
|
| 155 |
+
},
|
| 156 |
+
"engines": {
|
| 157 |
+
"node": ">=8"
|
| 158 |
+
}
|
| 159 |
+
},
|
| 160 |
+
"node_modules/typescript": {
|
| 161 |
+
"version": "5.9.3",
|
| 162 |
+
"resolved": "https://registry.npmjs.org/typescript/-/typescript-5.9.3.tgz",
|
| 163 |
+
"integrity": "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw==",
|
| 164 |
+
"dev": true,
|
| 165 |
+
"license": "Apache-2.0",
|
| 166 |
+
"bin": {
|
| 167 |
+
"tsc": "bin/tsc",
|
| 168 |
+
"tsserver": "bin/tsserver"
|
| 169 |
+
},
|
| 170 |
+
"engines": {
|
| 171 |
+
"node": ">=14.17"
|
| 172 |
+
}
|
| 173 |
+
},
|
| 174 |
+
"node_modules/undici-types": {
|
| 175 |
+
"version": "7.8.0",
|
| 176 |
+
"resolved": "https://registry.npmjs.org/undici-types/-/undici-types-7.8.0.tgz",
|
| 177 |
+
"integrity": "sha512-9UJ2xGDvQ43tYyVMpuHlsgApydB8ZKfVYTsLDhXkFL/6gfkp+U8xTGdh8pMJv1SpZna0zxG1DwsKZsreLbXBxw==",
|
| 178 |
+
"dev": true,
|
| 179 |
+
"license": "MIT"
|
| 180 |
+
},
|
| 181 |
+
"node_modules/ws": {
|
| 182 |
+
"version": "8.21.1",
|
| 183 |
+
"resolved": "https://registry.npmjs.org/ws/-/ws-8.21.1.tgz",
|
| 184 |
+
"integrity": "sha512-+0NTnW77fFN/DjQi6k/Sq/Yvk4Sgajw7urW8V+asjXnRgDs9gyGkdb7EzgfhA4goXsRIZKE28fzIXBHEzhuiWw==",
|
| 185 |
+
"license": "MIT",
|
| 186 |
+
"engines": {
|
| 187 |
+
"node": ">=10.0.0"
|
| 188 |
+
},
|
| 189 |
+
"peerDependencies": {
|
| 190 |
+
"bufferutil": "^4.0.1",
|
| 191 |
+
"utf-8-validate": ">=5.0.2"
|
| 192 |
+
},
|
| 193 |
+
"peerDependenciesMeta": {
|
| 194 |
+
"bufferutil": {
|
| 195 |
+
"optional": true
|
| 196 |
+
},
|
| 197 |
+
"utf-8-validate": {
|
| 198 |
+
"optional": true
|
| 199 |
+
}
|
| 200 |
+
}
|
| 201 |
+
},
|
| 202 |
+
"node_modules/zod": {
|
| 203 |
+
"version": "3.25.76",
|
| 204 |
+
"resolved": "https://registry.npmjs.org/zod/-/zod-3.25.76.tgz",
|
| 205 |
+
"integrity": "sha512-gzUt/qt81nXsFGKIFcC3YnfEAx5NkunCfnDlvuBSSFS02bcXu4Lmea0AFIUwbLWxWPx3d9p8S5QoaujKcNQxcQ==",
|
| 206 |
+
"license": "MIT",
|
| 207 |
+
"funding": {
|
| 208 |
+
"url": "https://github.com/sponsors/colinhacks"
|
| 209 |
+
}
|
| 210 |
+
},
|
| 211 |
+
"node_modules/zod-to-json-schema": {
|
| 212 |
+
"version": "3.25.2",
|
| 213 |
+
"resolved": "https://registry.npmjs.org/zod-to-json-schema/-/zod-to-json-schema-3.25.2.tgz",
|
| 214 |
+
"integrity": "sha512-O/PgfnpT1xKSDeQYSCfRI5Gy3hPf91mKVDuYLUHZJMiDFptvP41MSnWofm8dnCm0256ZNfZIM7DSzuSMAFnjHA==",
|
| 215 |
+
"license": "ISC",
|
| 216 |
+
"peerDependencies": {
|
| 217 |
+
"zod": "^3.25.28 || ^4"
|
| 218 |
+
}
|
| 219 |
+
}
|
| 220 |
+
}
|
| 221 |
+
}
|
integrations/btl3-native/package.json
ADDED
|
@@ -0,0 +1,17 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"name": "btl3-native-lmstudio",
|
| 3 |
+
"version": "0.1.0",
|
| 4 |
+
"private": true,
|
| 5 |
+
"scripts": {
|
| 6 |
+
"build": "tsc --noEmit",
|
| 7 |
+
"typecheck": "tsc --noEmit"
|
| 8 |
+
},
|
| 9 |
+
"dependencies": {
|
| 10 |
+
"@lmstudio/sdk": "1.5.0",
|
| 11 |
+
"openai": "6.48.0"
|
| 12 |
+
},
|
| 13 |
+
"devDependencies": {
|
| 14 |
+
"@types/node": "24.0.0",
|
| 15 |
+
"typescript": "5.9.3"
|
| 16 |
+
}
|
| 17 |
+
}
|
integrations/btl3-native/run-local
ADDED
|
@@ -0,0 +1,11 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/bin/sh
|
| 2 |
+
set -eu
|
| 3 |
+
|
| 4 |
+
root=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
|
| 5 |
+
if ! command -v lms >/dev/null 2>&1; then
|
| 6 |
+
echo "LM Studio CLI not found. Install it with: npx lmstudio install-cli" >&2
|
| 7 |
+
exit 2
|
| 8 |
+
fi
|
| 9 |
+
cd "$root"
|
| 10 |
+
npm ci --ignore-scripts
|
| 11 |
+
exec lms dev
|
integrations/btl3-native/src/config.ts
ADDED
|
@@ -0,0 +1,53 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import { createConfigSchematics } from "@lmstudio/sdk";
|
| 2 |
+
|
| 3 |
+
export const globalConfigSchematics = createConfigSchematics()
|
| 4 |
+
.field(
|
| 5 |
+
"baseUrl",
|
| 6 |
+
"string",
|
| 7 |
+
{
|
| 8 |
+
displayName: "BTL-3 server URL",
|
| 9 |
+
subtitle: "OpenAI-compatible endpoint exposed by the native BTL-3 runner.",
|
| 10 |
+
placeholder: "http://127.0.0.1:8080/v1",
|
| 11 |
+
},
|
| 12 |
+
"http://127.0.0.1:8080/v1",
|
| 13 |
+
)
|
| 14 |
+
.field(
|
| 15 |
+
"apiKey",
|
| 16 |
+
"string",
|
| 17 |
+
{
|
| 18 |
+
displayName: "Local API key",
|
| 19 |
+
subtitle: "Must match BTL3_API_KEY when authentication is enabled.",
|
| 20 |
+
isProtected: true,
|
| 21 |
+
},
|
| 22 |
+
"btl3-local",
|
| 23 |
+
)
|
| 24 |
+
.field(
|
| 25 |
+
"autoStart",
|
| 26 |
+
"boolean",
|
| 27 |
+
{
|
| 28 |
+
displayName: "Start the native runner automatically",
|
| 29 |
+
subtitle: "Launch BTL-3 locally when the configured endpoint is offline.",
|
| 30 |
+
},
|
| 31 |
+
true,
|
| 32 |
+
)
|
| 33 |
+
.field(
|
| 34 |
+
"runnerPath",
|
| 35 |
+
"string",
|
| 36 |
+
{
|
| 37 |
+
displayName: "Native runner path",
|
| 38 |
+
subtitle: "Optional. Defaults to BTL3_RUNNER_PATH or the installed BTL-3 runtime.",
|
| 39 |
+
placeholder: "/home/user/.local/share/btl3/libexec/llama-server",
|
| 40 |
+
},
|
| 41 |
+
"",
|
| 42 |
+
)
|
| 43 |
+
.field(
|
| 44 |
+
"modelPath",
|
| 45 |
+
"string",
|
| 46 |
+
{
|
| 47 |
+
displayName: "BTL-3 model path",
|
| 48 |
+
subtitle: "Optional. Defaults to BTL3_MODEL_PATH or the installed compact GGUF.",
|
| 49 |
+
placeholder: "/home/user/.local/share/btl3/model/BTL-3-Compact-AVQ2.gguf",
|
| 50 |
+
},
|
| 51 |
+
"",
|
| 52 |
+
)
|
| 53 |
+
.build();
|
integrations/btl3-native/src/generator.ts
ADDED
|
@@ -0,0 +1,185 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import {
|
| 2 |
+
type Chat,
|
| 3 |
+
type GeneratorController,
|
| 4 |
+
type InferParsedConfig,
|
| 5 |
+
type PluginContext,
|
| 6 |
+
} from "@lmstudio/sdk";
|
| 7 |
+
import OpenAI from "openai";
|
| 8 |
+
import type {
|
| 9 |
+
ChatCompletionChunk,
|
| 10 |
+
ChatCompletionMessageParam,
|
| 11 |
+
ChatCompletionMessageToolCall,
|
| 12 |
+
ChatCompletionTool,
|
| 13 |
+
} from "openai/resources/chat/completions";
|
| 14 |
+
import { globalConfigSchematics } from "./config";
|
| 15 |
+
import { ensureNativeRunner } from "./nativeRunner";
|
| 16 |
+
|
| 17 |
+
type Config = InferParsedConfig<typeof globalConfigSchematics>;
|
| 18 |
+
type ToolState = {
|
| 19 |
+
id: string;
|
| 20 |
+
name: string;
|
| 21 |
+
argumentFragments: string[];
|
| 22 |
+
};
|
| 23 |
+
|
| 24 |
+
function messages(history: Chat): ChatCompletionMessageParam[] {
|
| 25 |
+
const result: ChatCompletionMessageParam[] = [];
|
| 26 |
+
for (const message of history) {
|
| 27 |
+
switch (message.getRole()) {
|
| 28 |
+
case "system":
|
| 29 |
+
result.push({ role: "system", content: message.getText() });
|
| 30 |
+
break;
|
| 31 |
+
case "user":
|
| 32 |
+
result.push({ role: "user", content: message.getText() });
|
| 33 |
+
break;
|
| 34 |
+
case "assistant": {
|
| 35 |
+
const toolCalls: ChatCompletionMessageToolCall[] = message
|
| 36 |
+
.getToolCallRequests()
|
| 37 |
+
.map(call => ({
|
| 38 |
+
id: call.id ?? "",
|
| 39 |
+
type: "function",
|
| 40 |
+
function: {
|
| 41 |
+
name: call.name,
|
| 42 |
+
arguments: JSON.stringify(call.arguments ?? {}),
|
| 43 |
+
},
|
| 44 |
+
}));
|
| 45 |
+
result.push({
|
| 46 |
+
role: "assistant",
|
| 47 |
+
content: message.getText(),
|
| 48 |
+
...(toolCalls.length > 0 ? { tool_calls: toolCalls } : {}),
|
| 49 |
+
});
|
| 50 |
+
break;
|
| 51 |
+
}
|
| 52 |
+
case "tool":
|
| 53 |
+
for (const toolResult of message.getToolCallResults()) {
|
| 54 |
+
result.push({
|
| 55 |
+
role: "tool",
|
| 56 |
+
tool_call_id: toolResult.toolCallId ?? "",
|
| 57 |
+
content: toolResult.content,
|
| 58 |
+
});
|
| 59 |
+
}
|
| 60 |
+
break;
|
| 61 |
+
}
|
| 62 |
+
}
|
| 63 |
+
return result;
|
| 64 |
+
}
|
| 65 |
+
|
| 66 |
+
function tools(ctl: GeneratorController): ChatCompletionTool[] | undefined {
|
| 67 |
+
const result = ctl.getToolDefinitions().map<ChatCompletionTool>(tool => ({
|
| 68 |
+
type: "function",
|
| 69 |
+
function: {
|
| 70 |
+
name: tool.function.name,
|
| 71 |
+
description: tool.function.description,
|
| 72 |
+
parameters: tool.function.parameters ?? {},
|
| 73 |
+
},
|
| 74 |
+
}));
|
| 75 |
+
return result.length > 0 ? result : undefined;
|
| 76 |
+
}
|
| 77 |
+
|
| 78 |
+
function asError(error: unknown): Error {
|
| 79 |
+
return error instanceof Error ? error : new Error(String(error));
|
| 80 |
+
}
|
| 81 |
+
|
| 82 |
+
function finishTool(ctl: GeneratorController, state: ToolState): void {
|
| 83 |
+
ctl.toolCallGenerationStarted({ toolCallId: state.id });
|
| 84 |
+
if (state.name) {
|
| 85 |
+
ctl.toolCallGenerationNameReceived(state.name);
|
| 86 |
+
}
|
| 87 |
+
for (const fragment of state.argumentFragments) {
|
| 88 |
+
ctl.toolCallGenerationArgumentFragmentGenerated(fragment);
|
| 89 |
+
}
|
| 90 |
+
try {
|
| 91 |
+
const parsed = JSON.parse(
|
| 92 |
+
state.argumentFragments.join("") || "{}",
|
| 93 |
+
) as Record<string, unknown>;
|
| 94 |
+
ctl.toolCallGenerationEnded({
|
| 95 |
+
type: "function",
|
| 96 |
+
id: state.id,
|
| 97 |
+
name: state.name,
|
| 98 |
+
arguments: parsed,
|
| 99 |
+
});
|
| 100 |
+
} catch (error) {
|
| 101 |
+
ctl.toolCallGenerationFailed(asError(error));
|
| 102 |
+
}
|
| 103 |
+
}
|
| 104 |
+
|
| 105 |
+
function bufferToolDelta(
|
| 106 |
+
pending: Map<number, ToolState>,
|
| 107 |
+
call: ChatCompletionChunk.Choice.Delta.ToolCall,
|
| 108 |
+
): void {
|
| 109 |
+
const state = pending.get(call.index) ?? {
|
| 110 |
+
id: call.id ?? `call_${call.index}`,
|
| 111 |
+
name: "",
|
| 112 |
+
argumentFragments: [],
|
| 113 |
+
};
|
| 114 |
+
if (call.id) state.id = call.id;
|
| 115 |
+
if (call.function?.name) {
|
| 116 |
+
state.name += call.function.name;
|
| 117 |
+
}
|
| 118 |
+
if (call.function?.arguments) {
|
| 119 |
+
state.argumentFragments.push(call.function.arguments);
|
| 120 |
+
}
|
| 121 |
+
pending.set(call.index, state);
|
| 122 |
+
}
|
| 123 |
+
|
| 124 |
+
async function generate(
|
| 125 |
+
ctl: GeneratorController,
|
| 126 |
+
history: Chat,
|
| 127 |
+
config: Config,
|
| 128 |
+
): Promise<void> {
|
| 129 |
+
if (config.get("autoStart")) {
|
| 130 |
+
await ensureNativeRunner({
|
| 131 |
+
baseUrl: config.get("baseUrl"),
|
| 132 |
+
runnerPath: config.get("runnerPath"),
|
| 133 |
+
modelPath: config.get("modelPath"),
|
| 134 |
+
});
|
| 135 |
+
}
|
| 136 |
+
const client = new OpenAI({
|
| 137 |
+
apiKey: config.get("apiKey") || "btl3-local",
|
| 138 |
+
baseURL: config.get("baseUrl"),
|
| 139 |
+
});
|
| 140 |
+
const toolDefinitions = tools(ctl);
|
| 141 |
+
const pending = new Map<number, ToolState>();
|
| 142 |
+
try {
|
| 143 |
+
ctl.abortSignal.throwIfAborted();
|
| 144 |
+
const stream = await client.chat.completions.create(
|
| 145 |
+
{
|
| 146 |
+
model: "BTL-3",
|
| 147 |
+
messages: messages(history),
|
| 148 |
+
tools: toolDefinitions,
|
| 149 |
+
...(toolDefinitions ? { parallel_tool_calls: true } : {}),
|
| 150 |
+
stream: true,
|
| 151 |
+
},
|
| 152 |
+
{ signal: ctl.abortSignal },
|
| 153 |
+
);
|
| 154 |
+
for await (const chunk of stream) {
|
| 155 |
+
ctl.abortSignal.throwIfAborted();
|
| 156 |
+
const delta = chunk.choices[0]?.delta;
|
| 157 |
+
if (!delta) continue;
|
| 158 |
+
const reasoning = (
|
| 159 |
+
delta as typeof delta & { reasoning_content?: string }
|
| 160 |
+
).reasoning_content;
|
| 161 |
+
if (reasoning) {
|
| 162 |
+
ctl.fragmentGenerated(reasoning, { reasoningType: "reasoning" });
|
| 163 |
+
}
|
| 164 |
+
if (delta.content) {
|
| 165 |
+
ctl.fragmentGenerated(delta.content);
|
| 166 |
+
}
|
| 167 |
+
for (const call of delta.tool_calls ?? []) {
|
| 168 |
+
bufferToolDelta(pending, call);
|
| 169 |
+
}
|
| 170 |
+
}
|
| 171 |
+
for (const [, state] of [...pending].sort(([a], [b]) => a - b)) {
|
| 172 |
+
finishTool(ctl, state);
|
| 173 |
+
}
|
| 174 |
+
} catch (error) {
|
| 175 |
+
throw asError(error);
|
| 176 |
+
}
|
| 177 |
+
}
|
| 178 |
+
|
| 179 |
+
export async function main(context: PluginContext): Promise<void> {
|
| 180 |
+
context.withGlobalConfigSchematics(globalConfigSchematics);
|
| 181 |
+
context.withGenerator(async (ctl, history) => {
|
| 182 |
+
const config = ctl.getGlobalPluginConfig(globalConfigSchematics);
|
| 183 |
+
await generate(ctl, history, config);
|
| 184 |
+
});
|
| 185 |
+
}
|
integrations/btl3-native/src/index.ts
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
export { main } from "./generator";
|
integrations/btl3-native/src/nativeRunner.ts
ADDED
|
@@ -0,0 +1,120 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import { spawn, type ChildProcess } from "node:child_process";
|
| 2 |
+
import { homedir } from "node:os";
|
| 3 |
+
import { dirname, join } from "node:path";
|
| 4 |
+
|
| 5 |
+
type RunnerOptions = {
|
| 6 |
+
baseUrl: string;
|
| 7 |
+
runnerPath: string;
|
| 8 |
+
modelPath: string;
|
| 9 |
+
};
|
| 10 |
+
|
| 11 |
+
let child: ChildProcess | undefined;
|
| 12 |
+
let startup: Promise<void> | undefined;
|
| 13 |
+
let stderrTail = "";
|
| 14 |
+
let childError: Error | undefined;
|
| 15 |
+
|
| 16 |
+
function installedPaths(): { runner: string; model: string } {
|
| 17 |
+
if (process.platform === "win32") {
|
| 18 |
+
const root = process.env.LOCALAPPDATA ?? join(homedir(), "AppData", "Local");
|
| 19 |
+
return {
|
| 20 |
+
runner: join(root, "BTL3", "libexec", "llama-server.exe"),
|
| 21 |
+
model: join(root, "BTL3", "model", "BTL-3-Compact-AVQ2.gguf"),
|
| 22 |
+
};
|
| 23 |
+
}
|
| 24 |
+
const root = process.env.XDG_DATA_HOME ?? join(homedir(), ".local", "share");
|
| 25 |
+
return {
|
| 26 |
+
runner: join(root, "btl3", "libexec", "llama-server"),
|
| 27 |
+
model: join(root, "btl3", "model", "BTL-3-Compact-AVQ2.gguf"),
|
| 28 |
+
};
|
| 29 |
+
}
|
| 30 |
+
|
| 31 |
+
function endpoint(baseUrl: string, path: string): URL {
|
| 32 |
+
const url = new URL(baseUrl);
|
| 33 |
+
url.pathname = path;
|
| 34 |
+
url.search = "";
|
| 35 |
+
return url;
|
| 36 |
+
}
|
| 37 |
+
|
| 38 |
+
async function healthy(baseUrl: string): Promise<boolean> {
|
| 39 |
+
try {
|
| 40 |
+
const response = await fetch(endpoint(baseUrl, "/health"), {
|
| 41 |
+
signal: AbortSignal.timeout(1_500),
|
| 42 |
+
});
|
| 43 |
+
return response.ok;
|
| 44 |
+
} catch {
|
| 45 |
+
return false;
|
| 46 |
+
}
|
| 47 |
+
}
|
| 48 |
+
|
| 49 |
+
function runnerArguments(baseUrl: string, modelPath: string): string[] {
|
| 50 |
+
const url = new URL(baseUrl);
|
| 51 |
+
const port = url.port || "8080";
|
| 52 |
+
return [
|
| 53 |
+
"--model", modelPath,
|
| 54 |
+
"--host", url.hostname,
|
| 55 |
+
"--port", port,
|
| 56 |
+
"--no-webui",
|
| 57 |
+
"--offline",
|
| 58 |
+
"-np", "1",
|
| 59 |
+
];
|
| 60 |
+
}
|
| 61 |
+
|
| 62 |
+
async function waitUntilReady(baseUrl: string): Promise<void> {
|
| 63 |
+
const deadline = Date.now() + 60_000;
|
| 64 |
+
while (Date.now() < deadline) {
|
| 65 |
+
if (await healthy(baseUrl)) return;
|
| 66 |
+
if (childError) {
|
| 67 |
+
throw new Error(`BTL-3 runner failed to start: ${childError.message}`);
|
| 68 |
+
}
|
| 69 |
+
if (child?.exitCode !== null && child?.exitCode !== undefined) {
|
| 70 |
+
throw new Error(`BTL-3 runner exited (${child.exitCode}): ${stderrTail}`);
|
| 71 |
+
}
|
| 72 |
+
await new Promise(resolve => setTimeout(resolve, 250));
|
| 73 |
+
}
|
| 74 |
+
throw new Error(`BTL-3 runner did not become ready: ${stderrTail}`);
|
| 75 |
+
}
|
| 76 |
+
|
| 77 |
+
async function start(options: RunnerOptions): Promise<void> {
|
| 78 |
+
if (await healthy(options.baseUrl)) return;
|
| 79 |
+
const defaults = installedPaths();
|
| 80 |
+
const runner = options.runnerPath || process.env.BTL3_RUNNER_PATH || defaults.runner;
|
| 81 |
+
const model = options.modelPath || process.env.BTL3_MODEL_PATH || defaults.model;
|
| 82 |
+
const root = dirname(dirname(runner));
|
| 83 |
+
const libraryPath = join(root, "lib");
|
| 84 |
+
const env = { ...process.env };
|
| 85 |
+
if (process.platform === "win32") {
|
| 86 |
+
env.PATH = `${libraryPath};${env.PATH ?? ""}`;
|
| 87 |
+
env.GGML_BACKEND_PATH = join(libraryPath, "ggml-cuda.dll");
|
| 88 |
+
} else {
|
| 89 |
+
env.LD_LIBRARY_PATH =
|
| 90 |
+
`${libraryPath}${env.LD_LIBRARY_PATH ? `:${env.LD_LIBRARY_PATH}` : ""}`;
|
| 91 |
+
env.GGML_BACKEND_PATH = join(libraryPath, "libggml-cuda.so");
|
| 92 |
+
}
|
| 93 |
+
stderrTail = "";
|
| 94 |
+
childError = undefined;
|
| 95 |
+
child = spawn(runner, runnerArguments(options.baseUrl, model), {
|
| 96 |
+
windowsHide: true,
|
| 97 |
+
stdio: ["ignore", "ignore", "pipe"],
|
| 98 |
+
env,
|
| 99 |
+
});
|
| 100 |
+
child.stderr?.on("data", data => {
|
| 101 |
+
stderrTail = (stderrTail + String(data)).slice(-4_096);
|
| 102 |
+
});
|
| 103 |
+
child.once("error", error => {
|
| 104 |
+
childError = error;
|
| 105 |
+
stderrTail = (stderrTail + error.message).slice(-4_096);
|
| 106 |
+
});
|
| 107 |
+
await waitUntilReady(options.baseUrl);
|
| 108 |
+
}
|
| 109 |
+
|
| 110 |
+
export async function ensureNativeRunner(options: RunnerOptions): Promise<void> {
|
| 111 |
+
if (await healthy(options.baseUrl)) return;
|
| 112 |
+
startup ??= start(options).finally(() => {
|
| 113 |
+
startup = undefined;
|
| 114 |
+
});
|
| 115 |
+
await startup;
|
| 116 |
+
}
|
| 117 |
+
|
| 118 |
+
process.once("exit", () => {
|
| 119 |
+
child?.kill();
|
| 120 |
+
});
|
integrations/btl3-native/tsconfig.json
ADDED
|
@@ -0,0 +1,14 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"compilerOptions": {
|
| 3 |
+
"strict": true,
|
| 4 |
+
"module": "CommonJS",
|
| 5 |
+
"target": "ES2021",
|
| 6 |
+
"declaration": true,
|
| 7 |
+
"noImplicitOverride": true,
|
| 8 |
+
"esModuleInterop": true,
|
| 9 |
+
"skipLibCheck": true,
|
| 10 |
+
"rootDir": "src",
|
| 11 |
+
"outDir": "dist"
|
| 12 |
+
},
|
| 13 |
+
"include": ["src/**/*.ts"]
|
| 14 |
+
}
|
integrations/ollama/Modelfile
ADDED
|
@@ -0,0 +1,5 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Requires the explicitly labeled BTL-3 Patched Ollama distribution.
|
| 2 |
+
FROM ./BTL-3-Compact-AVQ2.gguf
|
| 3 |
+
PARAMETER num_ctx 32768
|
| 4 |
+
PARAMETER temperature 0.6
|
| 5 |
+
PARAMETER top_p 0.95
|
licenses/LICENSE
ADDED
|
@@ -0,0 +1,177 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
|
| 2 |
+
Apache License
|
| 3 |
+
Version 2.0, January 2004
|
| 4 |
+
http://www.apache.org/licenses/
|
| 5 |
+
|
| 6 |
+
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
| 7 |
+
|
| 8 |
+
1. Definitions.
|
| 9 |
+
|
| 10 |
+
"License" shall mean the terms and conditions for use, reproduction,
|
| 11 |
+
and distribution as defined by Sections 1 through 9 of this document.
|
| 12 |
+
|
| 13 |
+
"Licensor" shall mean the copyright owner or entity authorized by
|
| 14 |
+
the copyright owner that is granting the License.
|
| 15 |
+
|
| 16 |
+
"Legal Entity" shall mean the union of the acting entity and all
|
| 17 |
+
other entities that control, are controlled by, or are under common
|
| 18 |
+
control with that entity. For the purposes of this definition,
|
| 19 |
+
"control" means (i) the power, direct or indirect, to cause the
|
| 20 |
+
direction or management of such entity, whether by contract or
|
| 21 |
+
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
| 22 |
+
outstanding shares, or (iii) beneficial ownership of such entity.
|
| 23 |
+
|
| 24 |
+
"You" (or "Your") shall mean an individual or Legal Entity
|
| 25 |
+
exercising permissions granted by this License.
|
| 26 |
+
|
| 27 |
+
"Source" form shall mean the preferred form for making modifications,
|
| 28 |
+
including but not limited to software source code, documentation
|
| 29 |
+
source, and configuration files.
|
| 30 |
+
|
| 31 |
+
"Object" form shall mean any form resulting from mechanical
|
| 32 |
+
transformation or translation of a Source form, including but
|
| 33 |
+
not limited to compiled object code, generated documentation,
|
| 34 |
+
and conversions to other media types.
|
| 35 |
+
|
| 36 |
+
"Work" shall mean the work of authorship, whether in Source or
|
| 37 |
+
Object form, made available under the License, as indicated by a
|
| 38 |
+
copyright notice that is included in or attached to the work
|
| 39 |
+
(an example is provided in the Appendix below).
|
| 40 |
+
|
| 41 |
+
"Derivative Works" shall mean any work, whether in Source or Object
|
| 42 |
+
form, that is based on (or derived from) the Work and for which the
|
| 43 |
+
editorial revisions, annotations, elaborations, or other modifications
|
| 44 |
+
represent, as a whole, an original work of authorship. For the purposes
|
| 45 |
+
of this License, Derivative Works shall not include works that remain
|
| 46 |
+
separable from, or merely link (or bind by name) to the interfaces of,
|
| 47 |
+
the Work and Derivative Works thereof.
|
| 48 |
+
|
| 49 |
+
"Contribution" shall mean any work of authorship, including
|
| 50 |
+
the original version of the Work and any modifications or additions
|
| 51 |
+
to that Work or Derivative Works thereof, that is intentionally
|
| 52 |
+
submitted to Licensor for inclusion in the Work by the copyright owner
|
| 53 |
+
or by an individual or Legal Entity authorized to submit on behalf of
|
| 54 |
+
the copyright owner. For the purposes of this definition, "submitted"
|
| 55 |
+
means any form of electronic, verbal, or written communication sent
|
| 56 |
+
to the Licensor or its representatives, including but not limited to
|
| 57 |
+
communication on electronic mailing lists, source code control systems,
|
| 58 |
+
and issue tracking systems that are managed by, or on behalf of, the
|
| 59 |
+
Licensor for the purpose of discussing and improving the Work, but
|
| 60 |
+
excluding communication that is conspicuously marked or otherwise
|
| 61 |
+
designated in writing by the copyright owner as "Not a Contribution."
|
| 62 |
+
|
| 63 |
+
"Contributor" shall mean Licensor and any individual or Legal Entity
|
| 64 |
+
on behalf of whom a Contribution has been received by Licensor and
|
| 65 |
+
subsequently incorporated within the Work.
|
| 66 |
+
|
| 67 |
+
2. Grant of Copyright License. Subject to the terms and conditions of
|
| 68 |
+
this License, each Contributor hereby grants to You a perpetual,
|
| 69 |
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
| 70 |
+
copyright license to reproduce, prepare Derivative Works of,
|
| 71 |
+
publicly display, publicly perform, sublicense, and distribute the
|
| 72 |
+
Work and such Derivative Works in Source or Object form.
|
| 73 |
+
|
| 74 |
+
3. Grant of Patent License. Subject to the terms and conditions of
|
| 75 |
+
this License, each Contributor hereby grants to You a perpetual,
|
| 76 |
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
| 77 |
+
(except as stated in this section) patent license to make, have made,
|
| 78 |
+
use, offer to sell, sell, import, and otherwise transfer the Work,
|
| 79 |
+
where such license applies only to those patent claims licensable
|
| 80 |
+
by such Contributor that are necessarily infringed by their
|
| 81 |
+
Contribution(s) alone or by combination of their Contribution(s)
|
| 82 |
+
with the Work to which such Contribution(s) was submitted. If You
|
| 83 |
+
institute patent litigation against any entity (including a
|
| 84 |
+
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
| 85 |
+
or a Contribution incorporated within the Work constitutes direct
|
| 86 |
+
or contributory patent infringement, then any patent licenses
|
| 87 |
+
granted to You under this License for that Work shall terminate
|
| 88 |
+
as of the date such litigation is filed.
|
| 89 |
+
|
| 90 |
+
4. Redistribution. You may reproduce and distribute copies of the
|
| 91 |
+
Work or Derivative Works thereof in any medium, with or without
|
| 92 |
+
modifications, and in Source or Object form, provided that You
|
| 93 |
+
meet the following conditions:
|
| 94 |
+
|
| 95 |
+
(a) You must give any other recipients of the Work or
|
| 96 |
+
Derivative Works a copy of this License; and
|
| 97 |
+
|
| 98 |
+
(b) You must cause any modified files to carry prominent notices
|
| 99 |
+
stating that You changed the files; and
|
| 100 |
+
|
| 101 |
+
(c) You must retain, in the Source form of any Derivative Works
|
| 102 |
+
that You distribute, all copyright, patent, trademark, and
|
| 103 |
+
attribution notices from the Source form of the Work,
|
| 104 |
+
excluding those notices that do not pertain to any part of
|
| 105 |
+
the Derivative Works; and
|
| 106 |
+
|
| 107 |
+
(d) If the Work includes a "NOTICE" text file as part of its
|
| 108 |
+
distribution, then any Derivative Works that You distribute must
|
| 109 |
+
include a readable copy of the attribution notices contained
|
| 110 |
+
within such NOTICE file, excluding those notices that do not
|
| 111 |
+
pertain to any part of the Derivative Works, in at least one
|
| 112 |
+
of the following places: within a NOTICE text file distributed
|
| 113 |
+
as part of the Derivative Works; within the Source form or
|
| 114 |
+
documentation, if provided along with the Derivative Works; or,
|
| 115 |
+
within a display generated by the Derivative Works, if and
|
| 116 |
+
wherever such third-party notices normally appear. The contents
|
| 117 |
+
of the NOTICE file are for informational purposes only and
|
| 118 |
+
do not modify the License. You may add Your own attribution
|
| 119 |
+
notices within Derivative Works that You distribute, alongside
|
| 120 |
+
or as an addendum to the NOTICE text from the Work, provided
|
| 121 |
+
that such additional attribution notices cannot be construed
|
| 122 |
+
as modifying the License.
|
| 123 |
+
|
| 124 |
+
You may add Your own copyright statement to Your modifications and
|
| 125 |
+
may provide additional or different license terms and conditions
|
| 126 |
+
for use, reproduction, or distribution of Your modifications, or
|
| 127 |
+
for any such Derivative Works as a whole, provided Your use,
|
| 128 |
+
reproduction, and distribution of the Work otherwise complies with
|
| 129 |
+
the conditions stated in this License.
|
| 130 |
+
|
| 131 |
+
5. Submission of Contributions. Unless You explicitly state otherwise,
|
| 132 |
+
any Contribution intentionally submitted for inclusion in the Work
|
| 133 |
+
by You to the Licensor shall be under the terms and conditions of
|
| 134 |
+
this License, without any additional terms or conditions.
|
| 135 |
+
Notwithstanding the above, nothing herein shall supersede or modify
|
| 136 |
+
the terms of any separate license agreement you may have executed
|
| 137 |
+
with Licensor regarding such Contributions.
|
| 138 |
+
|
| 139 |
+
6. Trademarks. This License does not grant permission to use the trade
|
| 140 |
+
names, trademarks, service marks, or product names of the Licensor,
|
| 141 |
+
except as required for reasonable and customary use in describing the
|
| 142 |
+
origin of the Work and reproducing the content of the NOTICE file.
|
| 143 |
+
|
| 144 |
+
7. Disclaimer of Warranty. Unless required by applicable law or
|
| 145 |
+
agreed to in writing, Licensor provides the Work (and each
|
| 146 |
+
Contributor provides its Contributions) on an "AS IS" BASIS,
|
| 147 |
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
| 148 |
+
implied, including, without limitation, any warranties or conditions
|
| 149 |
+
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
| 150 |
+
PARTICULAR PURPOSE. You are solely responsible for determining the
|
| 151 |
+
appropriateness of using or redistributing the Work and assume any
|
| 152 |
+
risks associated with Your exercise of permissions under this License.
|
| 153 |
+
|
| 154 |
+
8. Limitation of Liability. In no event and under no legal theory,
|
| 155 |
+
whether in tort (including negligence), contract, or otherwise,
|
| 156 |
+
unless required by applicable law (such as deliberate and grossly
|
| 157 |
+
negligent acts) or agreed to in writing, shall any Contributor be
|
| 158 |
+
liable to You for damages, including any direct, indirect, special,
|
| 159 |
+
incidental, or consequential damages of any character arising as a
|
| 160 |
+
result of this License or out of the use or inability to use the
|
| 161 |
+
Work (including but not limited to damages for loss of goodwill,
|
| 162 |
+
work stoppage, computer failure or malfunction, or any and all
|
| 163 |
+
other commercial damages or losses), even if such Contributor
|
| 164 |
+
has been advised of the possibility of such damages.
|
| 165 |
+
|
| 166 |
+
9. Accepting Warranty or Additional Liability. While redistributing
|
| 167 |
+
the Work or Derivative Works thereof, You may choose to offer,
|
| 168 |
+
and charge a fee for, acceptance of support, warranty, indemnity,
|
| 169 |
+
or other liability obligations and/or rights consistent with this
|
| 170 |
+
License. However, in accepting such obligations, You may act only
|
| 171 |
+
on Your own behalf and on Your sole responsibility, not on behalf
|
| 172 |
+
of any other Contributor, and only if You agree to indemnify,
|
| 173 |
+
defend, and hold each Contributor harmless for any liability
|
| 174 |
+
incurred by, or claims asserted against, such Contributor by reason
|
| 175 |
+
of your accepting any such warranty or additional liability.
|
| 176 |
+
|
| 177 |
+
END OF TERMS AND CONDITIONS
|
licenses/LICENSE.runtime
ADDED
|
@@ -0,0 +1,22 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
MIT License
|
| 2 |
+
|
| 3 |
+
Copyright (c) 2026 Bad Theory Labs
|
| 4 |
+
|
| 5 |
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
| 6 |
+
of this software and associated documentation files (the "Software"), to deal
|
| 7 |
+
in the Software without restriction, including without limitation the rights
|
| 8 |
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
| 9 |
+
copies of the Software, and to permit persons to whom the Software is
|
| 10 |
+
furnished to do so, subject to the following conditions:
|
| 11 |
+
|
| 12 |
+
The above copyright notice and this permission notice shall be included in all
|
| 13 |
+
copies or substantial portions of the Software.
|
| 14 |
+
|
| 15 |
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
| 16 |
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
| 17 |
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
| 18 |
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
| 19 |
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
| 20 |
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
| 21 |
+
SOFTWARE.
|
| 22 |
+
|
licenses/THIRD_PARTY_NOTICES.md
ADDED
|
@@ -0,0 +1,30 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Third-party notices
|
| 2 |
+
|
| 3 |
+
## Qwen3.6-27B
|
| 4 |
+
|
| 5 |
+
BTL-3 is a post-trained adapter for `Qwen/Qwen3.6-27B`, pinned to revision
|
| 6 |
+
`6a9e13bd6fc8f0983b9b99948120bc37f49c13e9`. The base model is distributed
|
| 7 |
+
under Apache License 2.0. Users of the full-quality adapter must obtain the base
|
| 8 |
+
model separately and comply with its model card and license.
|
| 9 |
+
|
| 10 |
+
## Runtime dependencies
|
| 11 |
+
|
| 12 |
+
BTL-3 Compact's native runtime is derived from llama.cpp and links platform
|
| 13 |
+
libraries including Apple Accelerate/Metal or NVIDIA CUDA. The macOS bundle
|
| 14 |
+
includes its llama.cpp and OpenSSL notices; CUDA preview bundles include the
|
| 15 |
+
llama.cpp notice and NVIDIA redistributable runtime libraries. Those projects
|
| 16 |
+
retain their own copyrights and licenses and are not relicensed by Bad Theory
|
| 17 |
+
Labs.
|
| 18 |
+
|
| 19 |
+
## Benchmark artifacts
|
| 20 |
+
|
| 21 |
+
The evidence folder includes outputs and aggregate results from HumanEval and
|
| 22 |
+
BigCodeBench. It does not redistribute private evaluation cases, training
|
| 23 |
+
corpora, provider credentials, or paid-service configuration.
|
| 24 |
+
|
| 25 |
+
## Names and marks
|
| 26 |
+
|
| 27 |
+
Qwen, Hugging Face, PyTorch, Transformers, PEFT, vLLM, HumanEval,
|
| 28 |
+
BigCodeBench, LiveCodeBench, and BFCL are names of their respective projects
|
| 29 |
+
or owners. Their inclusion describes compatibility or evaluation and does not
|
| 30 |
+
imply endorsement.
|
tools/install_consumer_bundle.py
ADDED
|
@@ -0,0 +1,148 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Install a verified BTL-3 consumer runtime and its external model."""
|
| 2 |
+
|
| 3 |
+
from __future__ import annotations
|
| 4 |
+
|
| 5 |
+
import argparse
|
| 6 |
+
import hashlib
|
| 7 |
+
import json
|
| 8 |
+
import os
|
| 9 |
+
from pathlib import Path
|
| 10 |
+
import platform
|
| 11 |
+
import shutil
|
| 12 |
+
import tempfile
|
| 13 |
+
|
| 14 |
+
|
| 15 |
+
MODEL_NAME = "BTL-3-Compact-AVQ2.gguf"
|
| 16 |
+
MODEL_BYTES = 8_392_369_600
|
| 17 |
+
MODEL_SHA256 = "2ddf9527620a17a2a6739d184a7096c45712092e6589128792ec6254e94dc30c"
|
| 18 |
+
|
| 19 |
+
|
| 20 |
+
class InstallError(ValueError):
|
| 21 |
+
pass
|
| 22 |
+
|
| 23 |
+
|
| 24 |
+
def sha256(path: Path) -> str:
|
| 25 |
+
digest = hashlib.sha256()
|
| 26 |
+
with path.open("rb") as stream:
|
| 27 |
+
for chunk in iter(lambda: stream.read(4 * 1024 * 1024), b""):
|
| 28 |
+
digest.update(chunk)
|
| 29 |
+
return digest.hexdigest()
|
| 30 |
+
|
| 31 |
+
|
| 32 |
+
def default_prefix() -> Path:
|
| 33 |
+
if os.name == "nt":
|
| 34 |
+
root = os.environ.get("LOCALAPPDATA")
|
| 35 |
+
if not root:
|
| 36 |
+
raise InstallError("LOCALAPPDATA is unavailable; pass --prefix")
|
| 37 |
+
return Path(root) / "BTL3"
|
| 38 |
+
root = os.environ.get("XDG_DATA_HOME")
|
| 39 |
+
return Path(root) / "btl3" if root else Path.home() / ".local/share/btl3"
|
| 40 |
+
|
| 41 |
+
|
| 42 |
+
def read_manifest(runtime: Path) -> dict:
|
| 43 |
+
path = runtime / "bundle-manifest.json"
|
| 44 |
+
try:
|
| 45 |
+
manifest = json.loads(path.read_text())
|
| 46 |
+
except (OSError, json.JSONDecodeError) as error:
|
| 47 |
+
raise InstallError(f"invalid runtime manifest: {error}") from error
|
| 48 |
+
external = manifest.get("external_model", {})
|
| 49 |
+
expected = {
|
| 50 |
+
"filename": MODEL_NAME,
|
| 51 |
+
"bytes": MODEL_BYTES,
|
| 52 |
+
"sha256": MODEL_SHA256,
|
| 53 |
+
}
|
| 54 |
+
if any(external.get(key) != value for key, value in expected.items()):
|
| 55 |
+
raise InstallError("runtime expects a different BTL-3 model")
|
| 56 |
+
return manifest
|
| 57 |
+
|
| 58 |
+
|
| 59 |
+
def verify_runtime(runtime: Path, manifest: dict) -> None:
|
| 60 |
+
for name, expected in manifest.get("files", {}).items():
|
| 61 |
+
path = runtime / name
|
| 62 |
+
if "symlink" in expected:
|
| 63 |
+
if not path.is_symlink() or os.readlink(path) != expected["symlink"]:
|
| 64 |
+
raise InstallError(f"runtime symlink mismatch: {name}")
|
| 65 |
+
continue
|
| 66 |
+
if not path.is_file():
|
| 67 |
+
raise InstallError(f"runtime file is missing: {name}")
|
| 68 |
+
if path.stat().st_size != expected["bytes"]:
|
| 69 |
+
raise InstallError(f"runtime file size mismatch: {name}")
|
| 70 |
+
if sha256(path) != expected["sha256"]:
|
| 71 |
+
raise InstallError(f"runtime file checksum mismatch: {name}")
|
| 72 |
+
|
| 73 |
+
|
| 74 |
+
def verify_model(model: Path) -> None:
|
| 75 |
+
if not model.is_file():
|
| 76 |
+
raise InstallError(f"model is missing: {model}")
|
| 77 |
+
if model.stat().st_size != MODEL_BYTES:
|
| 78 |
+
raise InstallError(f"model size mismatch: {model.stat().st_size}")
|
| 79 |
+
if sha256(model) != MODEL_SHA256:
|
| 80 |
+
raise InstallError("model SHA-256 mismatch")
|
| 81 |
+
|
| 82 |
+
|
| 83 |
+
def install(runtime: Path, model: Path, prefix: Path, replace: bool) -> Path:
|
| 84 |
+
runtime, model, prefix = runtime.resolve(), model.resolve(), prefix.resolve()
|
| 85 |
+
manifest = read_manifest(runtime)
|
| 86 |
+
verify_runtime(runtime, manifest)
|
| 87 |
+
verify_model(model)
|
| 88 |
+
if prefix.exists() and not replace:
|
| 89 |
+
raise InstallError(f"install already exists (use --replace): {prefix}")
|
| 90 |
+
prefix.parent.mkdir(parents=True, exist_ok=True)
|
| 91 |
+
staging = Path(tempfile.mkdtemp(prefix=f".{prefix.name}-", dir=prefix.parent))
|
| 92 |
+
try:
|
| 93 |
+
shutil.copytree(runtime, staging, dirs_exist_ok=True, symlinks=True)
|
| 94 |
+
destination = staging / "model" / MODEL_NAME
|
| 95 |
+
destination.parent.mkdir(parents=True, exist_ok=True)
|
| 96 |
+
shutil.copy2(model, destination)
|
| 97 |
+
receipt = {
|
| 98 |
+
"schema_version": 1,
|
| 99 |
+
"installed_by": "BTL-3 consumer installer",
|
| 100 |
+
"platform": platform.platform(),
|
| 101 |
+
"runtime_target": manifest.get("target") or manifest.get("platform"),
|
| 102 |
+
"model": {
|
| 103 |
+
"path": f"model/{MODEL_NAME}",
|
| 104 |
+
"bytes": MODEL_BYTES,
|
| 105 |
+
"sha256": MODEL_SHA256,
|
| 106 |
+
},
|
| 107 |
+
}
|
| 108 |
+
(staging / "install-receipt.json").write_text(
|
| 109 |
+
json.dumps(receipt, indent=2) + "\n"
|
| 110 |
+
)
|
| 111 |
+
if prefix.exists():
|
| 112 |
+
backup = prefix.with_name(f".{prefix.name}.previous")
|
| 113 |
+
shutil.rmtree(backup, ignore_errors=True)
|
| 114 |
+
prefix.rename(backup)
|
| 115 |
+
try:
|
| 116 |
+
staging.rename(prefix)
|
| 117 |
+
except Exception:
|
| 118 |
+
backup.rename(prefix)
|
| 119 |
+
raise
|
| 120 |
+
shutil.rmtree(backup)
|
| 121 |
+
else:
|
| 122 |
+
staging.rename(prefix)
|
| 123 |
+
except Exception:
|
| 124 |
+
shutil.rmtree(staging, ignore_errors=True)
|
| 125 |
+
raise
|
| 126 |
+
return prefix
|
| 127 |
+
|
| 128 |
+
|
| 129 |
+
def main() -> None:
|
| 130 |
+
parser = argparse.ArgumentParser()
|
| 131 |
+
parser.add_argument("--runtime", type=Path, required=True)
|
| 132 |
+
parser.add_argument("--model", type=Path, required=True)
|
| 133 |
+
parser.add_argument("--prefix", type=Path, default=None)
|
| 134 |
+
parser.add_argument("--replace", action="store_true")
|
| 135 |
+
args = parser.parse_args()
|
| 136 |
+
try:
|
| 137 |
+
print(install(
|
| 138 |
+
args.runtime,
|
| 139 |
+
args.model,
|
| 140 |
+
args.prefix or default_prefix(),
|
| 141 |
+
args.replace,
|
| 142 |
+
))
|
| 143 |
+
except InstallError as error:
|
| 144 |
+
parser.error(str(error))
|
| 145 |
+
|
| 146 |
+
|
| 147 |
+
if __name__ == "__main__":
|
| 148 |
+
main()
|