Instructions to use Midasdf/AXELIDEA-QUON-14B-Japanese-v01-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use Midasdf/AXELIDEA-QUON-14B-Japanese-v01-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Midasdf/AXELIDEA-QUON-14B-Japanese-v01-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Midasdf/AXELIDEA-QUON-14B-Japanese-v01-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Midasdf/AXELIDEA-QUON-14B-Japanese-v01-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Midasdf/AXELIDEA-QUON-14B-Japanese-v01-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Midasdf/AXELIDEA-QUON-14B-Japanese-v01-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf Midasdf/AXELIDEA-QUON-14B-Japanese-v01-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Midasdf/AXELIDEA-QUON-14B-Japanese-v01-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Midasdf/AXELIDEA-QUON-14B-Japanese-v01-GGUF:Q4_K_M

Use Docker

docker model run hf.co/Midasdf/AXELIDEA-QUON-14B-Japanese-v01-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use Midasdf/AXELIDEA-QUON-14B-Japanese-v01-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Midasdf/AXELIDEA-QUON-14B-Japanese-v01-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Midasdf/AXELIDEA-QUON-14B-Japanese-v01-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Midasdf/AXELIDEA-QUON-14B-Japanese-v01-GGUF:Q4_K_M

Ollama
How to use Midasdf/AXELIDEA-QUON-14B-Japanese-v01-GGUF with Ollama:
```
ollama run hf.co/Midasdf/AXELIDEA-QUON-14B-Japanese-v01-GGUF:Q4_K_M
```
Unsloth Desktop
Docker Model Runner
How to use Midasdf/AXELIDEA-QUON-14B-Japanese-v01-GGUF with Docker Model Runner:
```
docker model run hf.co/Midasdf/AXELIDEA-QUON-14B-Japanese-v01-GGUF:Q4_K_M
```

Lemonade

How to use Midasdf/AXELIDEA-QUON-14B-Japanese-v01-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Midasdf/AXELIDEA-QUON-14B-Japanese-v01-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.AXELIDEA-QUON-14B-Japanese-v01-GGUF-Q4_K_M

List all available models

lemonade list

Atomic Chat

AXELIDEA-QUON-14B-Japanese-v01-GGUF

This repository contains an unofficial GGUF conversion of Axelidea/AXELIDEA-QUON-14B-Japanese-v01.

The source model is distributed under the MIT License. See LICENSE and the original model card for the authoritative license and model information.

File

AXELIDEA-QUON-14B-Japanese-v01-Q4_K_M.gguf
Quantization: Q4_K_M
File size: 8,890,306,528 bytes (about 8.89 GB)
Intended for llama.cpp-compatible applications such as LM Studio and Ollama

The original model uses a ChatML-style template. Applications should use the chat template embedded in the GGUF metadata when available.

Example with llama.cpp

llama-cli \
  -m AXELIDEA-QUON-14B-Japanese-v01-Q4_K_M.gguf \
  -c 8192 \
  -ngl 99 \
  -fa

On a 16 GB VRAM GPU, start with an 8,192-token context. A 16,384-token context may require reducing the context size, quantizing the KV cache, or offloading some layers to CPU depending on the backend and batch settings.

Conversion

The model was converted from the original BF16 safetensors checkpoint with llama.cpp's convert_hf_to_gguf.py, then quantized with llama-quantize using Q4_K_M.

Converter checkout: d7bd3bfcad3e29c7e49fd26f38c79ee3e9a3fd6b
Quantizer build: b9389-30af6e2b9
SHA256: b3ff6e0cb1d3f59f39b8bd410889d3d252a2189df7dfc5ee9d0c34e2922caf58

Validation

All six source safetensors shards matched the expected byte sizes from the original Hugging Face repository.
llama-cli loaded the Q4_K_M file with all layers offloaded to an AMD Radeon RX 9070 XT (16 GB VRAM) and generated Japanese text with an 8,192-token context and Flash Attention enabled.
A 16,384-token context load test also completed successfully on that GPU.

The converter reported Unknown RoPE type: default while processing the source configuration, which has rope_scaling: null; it exported rope scaling as NONE. The resulting GGUF passed the inference smoke tests above.

Attribution

Please cite and credit the original model authors when using this conversion:

Axelidea Inc., AXELIDEA-QUON-14B-Japanese-v01
https://huggingface.co/Axelidea/AXELIDEA-QUON-14B-Japanese-v01

Downloads last month: 123

GGUF

Model size

15B params

Architecture

llama

Hardware compatibility

4-bit

Model tree for Midasdf/AXELIDEA-QUON-14B-Japanese-v01-GGUF

Base model

microsoft/phi-4

Adapter

Axelidea/AXELIDEA-QUON-14B-Japanese-v01

Quantized

(1)

this model