Instructions to use skyuu72/Llama-Quantara-Sentinel-8B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use skyuu72/Llama-Quantara-Sentinel-8B with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf skyuu72/Llama-Quantara-Sentinel-8B:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf skyuu72/Llama-Quantara-Sentinel-8B:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf skyuu72/Llama-Quantara-Sentinel-8B:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf skyuu72/Llama-Quantara-Sentinel-8B:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf skyuu72/Llama-Quantara-Sentinel-8B:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf skyuu72/Llama-Quantara-Sentinel-8B:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf skyuu72/Llama-Quantara-Sentinel-8B:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf skyuu72/Llama-Quantara-Sentinel-8B:Q4_K_M

Use Docker

docker model run hf.co/skyuu72/Llama-Quantara-Sentinel-8B:Q4_K_M

LM Studio
Jan

vLLM

How to use skyuu72/Llama-Quantara-Sentinel-8B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "skyuu72/Llama-Quantara-Sentinel-8B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "skyuu72/Llama-Quantara-Sentinel-8B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/skyuu72/Llama-Quantara-Sentinel-8B:Q4_K_M

Ollama
How to use skyuu72/Llama-Quantara-Sentinel-8B with Ollama:
```
ollama run hf.co/skyuu72/Llama-Quantara-Sentinel-8B:Q4_K_M
```
Unsloth Desktop
Docker Model Runner
How to use skyuu72/Llama-Quantara-Sentinel-8B with Docker Model Runner:
```
docker model run hf.co/skyuu72/Llama-Quantara-Sentinel-8B:Q4_K_M
```

Lemonade

How to use skyuu72/Llama-Quantara-Sentinel-8B with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull skyuu72/Llama-Quantara-Sentinel-8B:Q4_K_M

Run and chat with the model

lemonade run user.Llama-Quantara-Sentinel-8B-Q4_K_M

List all available models

lemonade list

Atomic Chat

Llama-Quantara-Sentinel-8B / README-local-install.md

skyuu72

v1.0 card, AUP, Modelfiles, evaluation evidence

27e800b verified 14 days ago

preview code

Raw

History Blame

2.63 kB

Quantara Sentinel v6 — local install

Two quants, pick by memory bandwidth rather than raw size:

file	size	for
`quantara-sentinel-v6.Q4_K_M.gguf`	4.92 GB	RTX 4060 laptop (8 GB VRAM) — fits entirely in VRAM
`quantara-sentinel-v6.Q8_0.gguf`	8.54 GB	Legion Go S / Z1E (32 GB unified) and desktops

The 4060 (~~256 GB/s GDDR6) runs Q4_K_M roughly 2x faster than the Go S (~~120 GB/s LPDDR5X) because it fits. The moment a model exceeds 8 GB the 4060 falls off a PCIe cliff and the Go S — unified memory, no cliff — wins outright. So: Q4_K_M on the 4060, Q8_0 on the Go S. The Go S is SteamOS, so Vulkan works properly but root is immutable — use a static llama.cpp build in $HOME or distrobox, never steamos-readonly disable.

Install

ollama create quantara-sentinel -f Modelfile.Q4_K_M
ollama run quantara-sentinel

The Modelfile is hand-written and that is deliberate. This model uses Cisco Foundation-Sec's native grammar (<|system|> / <|user|> / <|assistant|>, turns closing on <|end_of_text|>), NOT llama-3.1's. Unsloth reports "No Ollama template mapping found" for this base and writes no Modelfile at all. Ship the llama-3.1 template here and every local user gets the runaway output that cost this project two rebuilds.

Do not change the SYSTEM line. The model was trained under it and behaves differently without it.

What it is

Llama-Quantara-Sentinel-8B — a Quantara fine-tune of Cisco's Foundation-Sec-8B-Instruct, built on Llama 3.1. Defensive scope: it explains and fixes, it does not weaponize.

Verified before release

Safety: all 6 gates pass. 0 judged weaponization leaks across 46 red-team probes, 100% refusal (regex and LLM judge agree), 0% over-refusal on both benign and dual-use, 0 runaway responses.
Terminates on its own native eos, unpatched.

Known weaknesses — read these

Phishing false negatives. 4 of 31 real phishing samples were cleared as legitimate (cap-0185, cap-0193, cap-0200, cap-0211). Do not use verdicts as the only gate on user-reported mail.
ATT&CK technique mis-mapping. It cites a real technique ID from the wrong tactic often enough to matter. Treat cited technique IDs as a starting point.
Canonical-fact grounding is uneven. Zerologon improved sharply this release (59.5% grounded, 2.4% fact errors) but BlueKeep regressed (66.7% / 12.8%). Verify vulnerability specifics against a primary source.
Anything time-sensitive — CVE details, current versions — should be checked against a live feed. Paste the advisory in rather than asking from memory.