Cree-Llama-3.3-70B 🦬

"ê-nîkânîhk ôma kîkway — This is a step forward."

Cree-Llama-3.3-70B is believed to be one of the first publicly available large language models fine-tuned specifically for the Cree language (Nêhiyawêwin / ᓀᐦᐃᔭᐍᐏᐣ). It is built on Meta's Llama 3.3 70B Instruct and supports both Cree syllabics (ᓀᐦᐃᔭᐍᐏᐣ) and Roman orthography (Nêhiyawêwin).

This is a first effort — a foundation. It is not perfect, and we know that. But every great journey begins with a single step, and this model represents a sincere commitment to ensuring that the Cree language has a place in the age of artificial intelligence.

---

🌱 Why This Matters

Cree (Nêhiyawêwin) is one of the most widely spoken Indigenous languages in Canada, with approximately 117,000 speakers across Alberta, Saskatchewan, Manitoba, Ontario, and Quebec. It has a rich oral tradition, a unique syllabic writing system developed in the 1840s, and a Roman orthography used widely today.

Despite its cultural significance, Cree — like most Indigenous languages — has been almost entirely absent from AI language models. Every major LLM has been trained overwhelmingly on English and a handful of other dominant languages. This project is an attempt to change that.

I, Micky Multani, am committed to continuing this work. This is Version 1. There will be a Version 2, a Version 3. With better data, better training, and community input, this model will improve. My goal is to make Cree language AI tools accessible and commonplace — for students, elders, educators, and anyone who wants to connect with this beautiful language.

---

📊 Model Details

Property	Value
Base model	Llama 3.3 70B Instruct
Fine-tuning method	QLoRA (4-bit, rank=32)
Training framework	Unsloth 2026.2.1
Custom Cree vocabulary	5,000 tokens added (133,256 total)
CPT training examples	445 Cree grammar/vocabulary examples
SFT training examples	21,141 Cree-English translation pairs
Quantization	Q4_K_M (GGUF)
File size	~40GB
Hardware trained on	NVIDIA A100 SXM4 80GB
Training time	~65 minutes total

---

🚀 How to Use

With LM Studio (Recommended for beginners)

1. Download the GGUF file
2. Open LM Studio
3. Load Cree-Llama-3.3-70B.Q4_K_M.gguf
4. Start chatting in Cree or English

With llama.cpp

./llama-cli \
    --model Cree-Llama-3.3-70B.Q4_K_M.gguf \
    --n-gpu-layers 99 \
    --ctx-size 1024 \
    --temp 0.7 \
    --prompt "Translate to Cree: How are you today?"

With Ollama

ollama run mickymultani/cree-llama-3.3-70b

---

💬 Example Outputs (Version 1)

These are real outputs from the current model. They are imperfect — and that is okay. This is the starting point.

English → Cree translation:

Prompt:  Translate to Cree: How are you today?
Output:  nîpîy cihkê

Cree → English translation:

Prompt:  Translate to English: tânisi kiya?
Output:  (model responds with English interpretation)

Syllabics:

Prompt:  What does ᑖᓂᓯ mean in English?
Output:  (model attempts to interpret syllabics)

---

🔧 Training Details

Stage 1: Custom Vocabulary Expansion

5,000 Cree-specific BPE tokens were trained on a Cree corpus and added to the base Llama tokenizer, expanding the vocabulary from 128,256 to 133,256 tokens. This reduced tokenization fertility by approximately 43% on Cree text — meaning the model processes Cree far more efficiently than the base model would.

Stage 2: Continued Pre-Training (CPT)

• Purpose: Teach the model Cree grammar patterns and vocabulary structure
• Data: 445 Cree language examples
• Steps: 100 | Learning rate: 2e-4
• Final loss: ~3.16

Stage 3: Supervised Fine-Tuning (SFT)

• Purpose: Teach translation and instruction following in Cree
• Data: 21,141 Cree-English instruction pairs
• Steps: 500 | Learning rate: 5e-5
• Final loss: ~1.51

LoRA Configuration

r = 32
lora_alpha = 32
target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
                  "gate_proj", "up_proj", "down_proj"]
lora_dropout = 0
bias = "none"
use_gradient_checkpointing = "unsloth"

---

⚠️ Current Limitations & Known Issues

This is Version 1. We are being fully transparent about what works and what doesn't:

• Training data is limited — 21,141 translation pairs is a starting point, not a complete dataset. The model will improve significantly with more data.
• Translations are not always accurate — The model produces Cree-like outputs but may make grammatical errors or mix dialects. Always verify with a fluent speaker.
• Dialectal variation — Cree has many dialects (Plains Cree, Swampy Cree, Woods Cree, etc.). This model was trained primarily on Plains Cree (nêhiyawêwin).
• Syllabics support is partial — The model handles Roman orthography better than syllabics in its current state.
• English mixing — The model may occasionally produce English words in Cree responses.

---

🗺️ Roadmap — What's Next

This is an active project. Here is what we are working toward:

Version 2 (planned):

• Significantly expanded training dataset (100,000+ examples)
• Better syllabics support with dedicated syllabics training data
• Dialect-specific fine-tuning options
• Smaller quantized versions (Q2_K, Q3_K_M) for accessibility on consumer hardware

Version 3 (planned):

• Multi-dialect support
• Audio/speech integration for oral language preservation
• Collaboration with Cree language educators and communities
• Web-based interface for non-technical users

Long-term vision:

• Make Cree language AI tools as accessible as Google Translate
• Support other Indigenous languages (Ojibwe, Inuktitut, Michif, Dene)
• Open-source all training data and scripts for community contribution

---

📁 Repository Contents

File	Description
Cree-Llama-3.3-70B.Q4_K_M.gguf	Main model file — use this for inference
tokenizer.json	Expanded tokenizer with 5,000 Cree tokens
tokenizer_config.json	Tokenizer configuration
special_tokens_map.json	Special tokens mapping
train_and_export.py	Full training script for reproducibility
final_cree_tokens.txt	The 5,000 custom Cree BPE tokens

---

🤝 Contributing & Community

This project needs the Cree community. If you are:

• A Cree speaker who wants to contribute corrections or translations
• A researcher working on Indigenous language NLP
• An educator who wants to use this in a classroom
• A developer who wants to help improve the model

Please reach out. This model belongs to the community it serves.

---

👤 Creator

Micky Multani @mickymultani

"Technology should serve everyone — including communities whose languages have been left behind. This is my small contribution to a much larger cause."

---

📜 License

This model is built on Llama 3.3 and is subject to the Llama 3.3 Community License.

---

🙏 Acknowledgments

Dedicated to the Cree people and all Indigenous communities working tirelessly to preserve their languages for future generations. Language is identity. Language is memory. Language is life.

kinanâskomitin — Thank you.