Qwen3.8-27B-Humanlike-Chat-GGUF

Instructions to use 0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use 0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf 0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf 0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf 0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf 0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf 0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf 0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf 0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf 0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF:Q4_K_M

Use Docker

docker model run hf.co/0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use 0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF:Q4_K_M

Ollama
How to use 0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF with Ollama:
```
ollama run hf.co/0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF:Q4_K_M
```
Unsloth Desktop

How to use 0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf 0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use 0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF with Docker Model Runner:
```
docker model run hf.co/0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF:Q4_K_M
```

Lemonade

How to use 0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull 0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Qwen3.8-27B-Humanlike-Chat-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use 0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf 0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default 0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use 0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf 0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "0xSojalSec/Qwen3.8-27B-Humanlike-Chat-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Qwen3.8-27B-Humanlike-Chat-GGUF / SERVING.md

0xSojalSec

kvyb

Duplicate from LessThanThreeAI/Qwen3.8-27B-Humanlike-Chat-GGUF

f1f655f 11 days ago

preview code

Raw History Blame Contribute Delete

7.73 kB

Serving configuration: vLLM base plus step-576 LoRA

This file documents how the hosted Qwen3.8-27B Humanlike Chat endpoint is configured. It is the serving recipe behind https://api.lessthanthreeai.com/v1, not the local llama.cpp path described on the model card.

No private conversation content, credentials, or infrastructure identifiers are included.

Why this path exists

The GGUF release merges the adaptation into one file. vLLM serves the adapter dynamically instead: the abliterated parent stays frozen and the LoRA is mounted at startup. That keeps the base and adapter identities separate and verifiable, and it is the configuration behind the hosted endpoint named on the model card.

Serving the GGUF files through vLLM is possible only with the experimental vLLM GGUF plugin and was not validated here. The two paths are not interchangeable.

Artifacts

Component	Identity
Base	`huihui-ai/Huihui-Qwen3.8-27B-abliterated` at revision `d42ca8978c5a66e92c3446d46e8adfe03ef692ff`
Adapter	V3 step-576 LoRA, rank 256, training alpha 32, 992 tensors across 496 module pairs
Mounted strength	0.7, using serving `lora_alpha: 22.4` with unchanged rank and tensors
Served precision	BF16 derivative of the canonical V3 adapter, 3,735,445,032 bytes, sha256 `e9973c4dc65a2f6c1c3e138ce91b565cf3b51319312508e2663c9f164f8cd0e8`
Served model name	`qwen38-personal-step863` (legacy alias; actual checkpoint is step-576)

The adapter is mounted unmerged. Do not serve a checkpoint that already contains it. In this vLLM 0.27.1 setup, a separate serving copy of adapter_config.json sets lora_alpha to 32 * 0.7 = 22.4; the original configuration and weights stay unchanged. Apply the scale only once. This is startup configuration, not a supported per-request strength parameter on the hosted API.

vLLM configuration

Tested stack: runpod-workers/worker-vllm v2.25.1 with vLLM 0.27.1, Transformers 5.14.1, and PEFT 0.19.1. Live responses carry the engine fingerprint vllm-0.27.1-ad0ba41a, which is a quick way to confirm which serving stack answered a request.

These are the worker environment variables as deployed:

MODEL_NAME=/models/base
DTYPE=bfloat16
ENABLE_LORA=true
MAX_LORAS=1
MAX_CPU_LORAS=1
MAX_LORA_RANK=256
LORA_DTYPE=bfloat16
LORA_MODULES=qwen38-personal-step863=/models/adapter-bf16
OPENAI_SERVED_MODEL_NAME_OVERRIDE=qwen38-base-do-not-use
REQUIRED_MODEL=qwen38-personal-step863
REASONING_PARSER=qwen3
MAX_MODEL_LEN=131072
MAX_NUM_SEQS=3
MAX_CONCURRENCY=3
DEFAULT_MAX_TOKENS=8192
MAX_OUTPUT_TOKENS=50000
GPU_MEMORY_UTILIZATION=0.95
ENFORCE_EAGER=true
ATTENTION_BACKEND=TRITON_ATTN
VLLM_EXTRA_ARGS=--language-model-only --default-chat-template-kwargs '{"enable_thinking":false}' --gdn-prefill-backend triton
VLLM_USE_FLASHINFER_SAMPLER=0
VLLM_STARTUP_TIMEOUT=1800
REQUEST_TIMEOUT=3600
UVICORN_LOG_LEVEL=warning
ENABLE_LOG_REQUESTS=false
TOKENIZERS_PARALLELISM=false
HF_HUB_OFFLINE=1
TRANSFORMERS_OFFLINE=1
RAW_OPENAI_OUTPUT=1

Notes on the non-default settings:

LORA_MODULES uses the name=path form. The JSON-array form is rejected by this worker version with Invalid fields for --lora-modules.
OPENAI_SERVED_MODEL_NAME_OVERRIDE is set to a sentinel that clients will not use, so a request that fails to select the LoRA module errors instead of silently returning base-model output. The public alias is applied in front of the engine.
REQUIRED_MODEL, DEFAULT_MAX_TOKENS, MAX_OUTPUT_TOKENS, and RAW_OPENAI_OUTPUT are worker request-guard settings rather than engine flags.
VLLM_EXTRA_ARGS carries flags the worker does not expose as first-class variables, including --language-model-only, which drops vision and MTP/NextN tensors to match the text-only release.
VLLM_USE_FLASHINFER_SAMPLER=0 keeps sampling off the FlashInfer path on this stack.
HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE keep the worker from reaching the Hub at runtime. The engine receives no Hub or object-storage credentials.
MAX_MODEL_LEN is the combined prompt plus output ceiling, not the context window alone.
ENFORCE_EAGER=true and ATTENTION_BACKEND=TRITON_ATTN are the combination validated on this hardware.
MODEL_NAME and the LORA_MODULES path above are placeholders for the local staging layout; the base and adapter are staged on the worker's attached volume before startup.

The worker starts vLLM with approximately:

vllm serve --host 127.0.0.1 --port 8000 --model <base> --served-model-name qwen38-base-do-not-use \
  --enable-lora --lora-modules qwen38-personal-step863=<adapter> --max-loras 1 --max-lora-rank 256 \
  --lora-dtype bfloat16 --max-cpu-loras 1 --max-model-len 131072 --max-num-seqs 3 \
  --dtype bfloat16 --gpu-memory-utilization 0.95 --enforce-eager --attention-backend TRITON_ATTN \
  --reasoning-parser qwen3 --language-model-only --gdn-prefill-backend triton \
  --default-chat-template-kwargs '{"enable_thinking":false}' --uvicorn-log-level warning

Hosting shape

RunPod Serverless, one 96 GB NVIDIA RTX PRO 6000 Blackwell Server Edition, tensor parallel size 1.

Setting	Value
Minimum workers	0
Maximum workers	1
Standby worker records	1
Scaler	Request count, 1
Idle timeout	300 seconds
Execution timeout	3,600,000 ms
Engine concurrency	3
FlashBoot	Enabled

After the last request the worker stays warm for the idle window, then exits and stops GPU billing. A full cold start includes provider provisioning, base load, adapter load, and engine initialization. Measured full-cold latency was 154.9 seconds and a FlashBoot revival completed in 3.1 seconds. RunPod may keep one standby worker record after scale-down; the billing-relevant state is the underlying pod, and desiredStatus=EXITED means no active GPU compute.

Request contract

Setting	Value
Base URL	`https://api.lessthanthreeai.com/v1`
Model	`qwen3.8-27b-humanlike-chat`
API key	Not required
Default output	8,192 tokens
Maximum requested output	50,000 tokens
Prompt plus output ceiling	131,072 tokens
Recommended client timeout	At least 300 seconds

A streaming request that arrives at zero workers stays connected while the model starts. Some deployments emit an SSE comment of the form : status={"state":"model_starting",...} before the first data chunk; standard OpenAI SDKs ignore it, and a custom client should display it separately rather than appending it to the conversation.

Thinking is off by default and preserve_thinking=false is forced. Supported reasoning efforts are low, medium, and xhigh. Reasoning deltas arrive in the reasoning field and final text in content.

Three concurrent short requests are supported per worker; further requests queue. All requests share one KV cache, so three simultaneous maximum-length contexts are not supported.

Adapter availability

The BF16 step-576 adapter used here is not published on the Hub, so the configuration above cannot be reproduced end to end from public files alone. The published F32 GGUF LoRA is the step-863 adapter exported for llama.cpp (--lora), sha256 e4454617cc5262e0f33f1544ec89fba2873d37273a94b46f321a3e9a4b3bf019. It is a different checkpoint and a GGUF LoRA rather than a PEFT safetensors adapter. See the model card for its base-matching rules and starting-strength recommendation; parity with this hosted path is not established.

Evidence

Runtime and quantization characterization, including the matched first-turn diagnosis and the bounded serverless comparison, is in FINDINGS.md.