Configuration Parsing Warning:In config.json: "quantization_config.modules_to_not_convert" must be an array
Charlie 1.5 AWQ (12B, 4-bit)
Charlie 1.5 AWQ is a 4-bit Activation-Aware Quantized (AWQ) variant of the Charlie 1.5 12B-parameter large language model, optimized for high-throughput, low-VRAM inference while preserving reasoning quality close to the full BF16 model.
Built on the same Mistral-based architecture, this version is particularly well-suited for long-context enterprise workloads where memory efficiency and deployment cost are critical.
With support for a 131,072-token (128k+) context window, Charlie 1.5 AWQ enables native analysis of large documents—such as financial filings, contracts, reports, and codebases—without requiring retrieval-augmented generation (RAG) or external chunking systems.
Model Summary
| Attribute | Description |
|---|---|
| Base Model | Charlie 1.5 (BF16) |
| Quantization | 4-bit Activation-Aware Quantization (AWQ) |
| Architecture | Mistral-based decoder-only transformer |
| Parameters | ~12B |
| Layers | 40 |
| Hidden Size | 5,120 |
| Context Window | 131,072 tokens |
| Vocabulary Size | 131,072 |
| Model Size | ~8 GB |
| License | Apache License 2.0 |
Model Highlights
- AWQ Quantization: 4-bit activation-aware quantization with minimal accuracy degradation
- Improved Throughput: Up to ~2× faster inference compared to BF16 on supported GPUs
- Reduced VRAM Footprint: Significantly lower memory requirements for deployment
- Long-Context Support: Full 131k-token context window retained
- Enterprise-Friendly: Designed for cost-efficient inference on mid-range GPUs (e.g., NVIDIA A10G)
Performance & Benchmarks
| Benchmark | Score |
|---|---|
| MMLU | ~66 |
| MMLU-Pro | ~37 |
| ARC-Challenge | ~59 |
| HellaSwag | ~82 |
Inference Performance (NVIDIA A10G)
- Time to First Token (TTFT): ~80 ms
- Throughput: ~218 tokens/sec
- Quantization: 4-bit AWQ
Performance may vary depending on hardware, batch size, and prompt length.
Intended Use & Scope
Charlie 1.5 AWQ is intended for:
- High-throughput long-context inference
- Enterprise and production deployments with constrained VRAM
- Cost-efficient serving of large documents
- Commercial and non-commercial applications
- Downstream fine-tuning or adaptation workflows (with appropriate tooling)
The model is provided as-is and should be independently evaluated before use in high-risk or safety-critical applications.
Usage
Charlie 1.5 AWQ can be used with the Hugging Face transformers library and AWQ-compatible runtimes:
import torch
from transformers import pipeline
pipe = pipeline("text-generation",
model=repo_id,
device_map='auto',
torch_dtype=torch.float16)
prompt = """
Analyze the impact of a 15% tariff
increase on lithium-ion components from the Asia-Pacific region.
1. Identify the top 3 Tier 2 suppliers most at risk based on current lead times.
2. Propose a diversification strategy for our European assembly plant.
3. Calculate the projected shift in COGS if we pivot 40% of sourcing to Mexico.
"""
messages = [
{"role": "system", "content": ""},
{"role": "user", "content": prompt},
]
outputs = pipe(messages,
max_new_tokens=512,
do_sample=True,
temperature=0.7,
use_cache=True,
return_full_text=False,
num_return_sequences=1
)
for output in outputs:
print(output['generated_text'])
Technical Specifications
- Hidden Size: 5,120
- Intermediate Size: 14,336
- Attention Heads: 32
- KV Heads: 8 (Grouped Query Attention)
- Activation Function: SiLU
- Normalization: RMSNorm (epsilon: 1e-05)
- RoPE Theta: 1,000,000
- Max Position Embeddings: 131,072
- Quantization: AWQ, 4-bit (GEMM backend)
License
The model is released freely and without restriction under the Apache License 2.0. There are no restrictions on downstream usage beyond those stated in the license.
Citation & Attribution
If you use Charlie 1.5 in research or commercial applications, please attribute it to the original Gaudium AI development team.
- Downloads last month
- 27