Qwen3.8-27B-Humanlike-Chat-GGUF

Instructions to use 0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use 0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf 0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf 0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf 0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf 0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf 0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf 0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf 0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf 0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF:Q4_K_M

Use Docker

docker model run hf.co/0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use 0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF:Q4_K_M

Ollama
How to use 0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF with Ollama:
```
ollama run hf.co/0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF:Q4_K_M
```
Unsloth Desktop

How to use 0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf 0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use 0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF with Docker Model Runner:
```
docker model run hf.co/0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF:Q4_K_M
```

Lemonade

How to use 0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull 0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Qwen3.8-27B-Humanlike-Chat-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use 0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf 0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default 0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use 0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf 0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Qwen3.8-27B-Humanlike-Chat-GGUF / SERVING.md

0xSojalSec

kvyb

Duplicate from LessThanThreeAI/Qwen3.8-27B-Humanlike-Chat-GGUF

f1f655f 12 days ago

preview code

Raw History Blame Contribute Delete

7.73 kB

	# Serving configuration: vLLM base plus step-576 LoRA

	This file documents how the hosted Qwen3.8-27B Humanlike Chat endpoint is configured. It is the serving recipe behind `https://api.lessthanthreeai.com/v1`, not the local llama.cpp path described on the [model card](README.md).

	No private conversation content, credentials, or infrastructure identifiers are included.

	## Why this path exists

	The GGUF release merges the adaptation into one file. vLLM serves the adapter dynamically instead: the abliterated parent stays frozen and the LoRA is mounted at startup. That keeps the base and adapter identities separate and verifiable, and it is the configuration behind the hosted endpoint named on the model card.

	Serving the GGUF files through vLLM is possible only with the experimental vLLM GGUF plugin and was not validated here. The two paths are not interchangeable.

	## Artifacts

	\| Component \| Identity \|
	\|---\|---\|
	\| Base \| `huihui-ai/Huihui-Qwen3.8-27B-abliterated` at revision `d42ca8978c5a66e92c3446d46e8adfe03ef692ff` \|
	\| Adapter \| V3 step-576 LoRA, rank 256, training alpha 32, 992 tensors across 496 module pairs \|
	\| Mounted strength \| 0.7, using serving `lora_alpha: 22.4` with unchanged rank and tensors \|
	\| Served precision \| BF16 derivative of the canonical V3 adapter, 3,735,445,032 bytes, sha256 `e9973c4dc65a2f6c1c3e138ce91b565cf3b51319312508e2663c9f164f8cd0e8` \|
	\| Served model name \| `qwen38-personal-step863` (legacy alias; actual checkpoint is step-576) \|

	The adapter is mounted unmerged. Do not serve a checkpoint that already contains it. In this vLLM 0.27.1 setup, a separate serving copy of `adapter_config.json` sets `lora_alpha` to `32 * 0.7 = 22.4`; the original configuration and weights stay unchanged. Apply the scale only once. This is startup configuration, not a supported per-request strength parameter on the hosted API.

	## vLLM configuration

	Tested stack: `runpod-workers/worker-vllm` v2.25.1 with vLLM 0.27.1, Transformers 5.14.1, and PEFT 0.19.1. Live responses carry the engine fingerprint `vllm-0.27.1-ad0ba41a`, which is a quick way to confirm which serving stack answered a request.

	These are the worker environment variables as deployed:

	```text
	MODEL_NAME=/models/base
	DTYPE=bfloat16
	ENABLE_LORA=true
	MAX_LORAS=1
	MAX_CPU_LORAS=1
	MAX_LORA_RANK=256
	LORA_DTYPE=bfloat16
	LORA_MODULES=qwen38-personal-step863=/models/adapter-bf16
	OPENAI_SERVED_MODEL_NAME_OVERRIDE=qwen38-base-do-not-use
	REQUIRED_MODEL=qwen38-personal-step863
	REASONING_PARSER=qwen3
	MAX_MODEL_LEN=131072
	MAX_NUM_SEQS=3
	MAX_CONCURRENCY=3
	DEFAULT_MAX_TOKENS=8192
	MAX_OUTPUT_TOKENS=50000
	GPU_MEMORY_UTILIZATION=0.95
	ENFORCE_EAGER=true
	ATTENTION_BACKEND=TRITON_ATTN
	VLLM_EXTRA_ARGS=--language-model-only --default-chat-template-kwargs '{"enable_thinking":false}' --gdn-prefill-backend triton
	VLLM_USE_FLASHINFER_SAMPLER=0
	VLLM_STARTUP_TIMEOUT=1800
	REQUEST_TIMEOUT=3600
	UVICORN_LOG_LEVEL=warning
	ENABLE_LOG_REQUESTS=false
	TOKENIZERS_PARALLELISM=false
	HF_HUB_OFFLINE=1
	TRANSFORMERS_OFFLINE=1
	RAW_OPENAI_OUTPUT=1
	```

	Notes on the non-default settings:

	- `LORA_MODULES` uses the `name=path` form. The JSON-array form is rejected by this worker version with `Invalid fields for --lora-modules`.
	- `OPENAI_SERVED_MODEL_NAME_OVERRIDE` is set to a sentinel that clients will not use, so a request that fails to select the LoRA module errors instead of silently returning base-model output. The public alias is applied in front of the engine.
	- `REQUIRED_MODEL`, `DEFAULT_MAX_TOKENS`, `MAX_OUTPUT_TOKENS`, and `RAW_OPENAI_OUTPUT` are worker request-guard settings rather than engine flags.
	- `VLLM_EXTRA_ARGS` carries flags the worker does not expose as first-class variables, including `--language-model-only`, which drops vision and MTP/NextN tensors to match the text-only release.
	- `VLLM_USE_FLASHINFER_SAMPLER=0` keeps sampling off the FlashInfer path on this stack.
	- `HF_HUB_OFFLINE` and `TRANSFORMERS_OFFLINE` keep the worker from reaching the Hub at runtime. The engine receives no Hub or object-storage credentials.
	- `MAX_MODEL_LEN` is the combined prompt plus output ceiling, not the context window alone.
	- `ENFORCE_EAGER=true` and `ATTENTION_BACKEND=TRITON_ATTN` are the combination validated on this hardware.
	- `MODEL_NAME` and the `LORA_MODULES` path above are placeholders for the local staging layout; the base and adapter are staged on the worker's attached volume before startup.

	The worker starts vLLM with approximately:

	```text
	vllm serve --host 127.0.0.1 --port 8000 --model <base> --served-model-name qwen38-base-do-not-use \
	--enable-lora --lora-modules qwen38-personal-step863=<adapter> --max-loras 1 --max-lora-rank 256 \
	--lora-dtype bfloat16 --max-cpu-loras 1 --max-model-len 131072 --max-num-seqs 3 \
	--dtype bfloat16 --gpu-memory-utilization 0.95 --enforce-eager --attention-backend TRITON_ATTN \
	--reasoning-parser qwen3 --language-model-only --gdn-prefill-backend triton \
	--default-chat-template-kwargs '{"enable_thinking":false}' --uvicorn-log-level warning
	```

	## Hosting shape

	RunPod Serverless, one 96 GB NVIDIA RTX PRO 6000 Blackwell Server Edition, tensor parallel size 1.

	\| Setting \| Value \|
	\|---\|---\|
	\| Minimum workers \| 0 \|
	\| Maximum workers \| 1 \|
	\| Standby worker records \| 1 \|
	\| Scaler \| Request count, 1 \|
	\| Idle timeout \| 300 seconds \|
	\| Execution timeout \| 3,600,000 ms \|
	\| Engine concurrency \| 3 \|
	\| FlashBoot \| Enabled \|

	After the last request the worker stays warm for the idle window, then exits and stops GPU billing. A full cold start includes provider provisioning, base load, adapter load, and engine initialization. Measured full-cold latency was 154.9 seconds and a FlashBoot revival completed in 3.1 seconds. RunPod may keep one standby worker record after scale-down; the billing-relevant state is the underlying pod, and `desiredStatus=EXITED` means no active GPU compute.

	## Request contract

	\| Setting \| Value \|
	\|---\|---\|
	\| Base URL \| `https://api.lessthanthreeai.com/v1` \|
	\| Model \| `qwen3.8-27b-humanlike-chat` \|
	\| API key \| Not required \|
	\| Default output \| 8,192 tokens \|
	\| Maximum requested output \| 50,000 tokens \|
	\| Prompt plus output ceiling \| 131,072 tokens \|
	\| Recommended client timeout \| At least 300 seconds \|

	A streaming request that arrives at zero workers stays connected while the model starts. Some deployments emit an SSE comment of the form `: status={"state":"model_starting",...}` before the first data chunk; standard OpenAI SDKs ignore it, and a custom client should display it separately rather than appending it to the conversation.

	Thinking is off by default and `preserve_thinking=false` is forced. Supported reasoning efforts are `low`, `medium`, and `xhigh`. Reasoning deltas arrive in the `reasoning` field and final text in `content`.

	Three concurrent short requests are supported per worker; further requests queue. All requests share one KV cache, so three simultaneous maximum-length contexts are not supported.

	## Adapter availability

	The BF16 step-576 adapter used here is not published on the Hub, so the configuration above cannot be reproduced end to end from public files alone. The published [F32 GGUF LoRA](Qwen3.8-27B-Humanlike-Chat-Step863-LoRA-F32.gguf) is the step-863 adapter exported for llama.cpp (`--lora`), sha256 `e4454617cc5262e0f33f1544ec89fba2873d37273a94b46f321a3e9a4b3bf019`. It is a different checkpoint and a GGUF LoRA rather than a PEFT safetensors adapter. See the model card for its base-matching rules and starting-strength recommendation; parity with this hosted path is not established.

	## Evidence

	Runtime and quantization characterization, including the matched first-turn diagnosis and the bounded serverless comparison, is in [`FINDINGS.md`](FINDINGS.md).