Instructions to use konstantindobler/mistral7b-ar-tokenizer-swap-pure-bf16-anneal-ablation with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use konstantindobler/mistral7b-ar-tokenizer-swap-pure-bf16-anneal-ablation with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="konstantindobler/mistral7b-ar-tokenizer-swap-pure-bf16-anneal-ablation")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("konstantindobler/mistral7b-ar-tokenizer-swap-pure-bf16-anneal-ablation")
model = AutoModelForCausalLM.from_pretrained("konstantindobler/mistral7b-ar-tokenizer-swap-pure-bf16-anneal-ablation", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use konstantindobler/mistral7b-ar-tokenizer-swap-pure-bf16-anneal-ablation with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "konstantindobler/mistral7b-ar-tokenizer-swap-pure-bf16-anneal-ablation"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "konstantindobler/mistral7b-ar-tokenizer-swap-pure-bf16-anneal-ablation",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/konstantindobler/mistral7b-ar-tokenizer-swap-pure-bf16-anneal-ablation

SGLang

How to use konstantindobler/mistral7b-ar-tokenizer-swap-pure-bf16-anneal-ablation with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "konstantindobler/mistral7b-ar-tokenizer-swap-pure-bf16-anneal-ablation" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "konstantindobler/mistral7b-ar-tokenizer-swap-pure-bf16-anneal-ablation",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "konstantindobler/mistral7b-ar-tokenizer-swap-pure-bf16-anneal-ablation" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "konstantindobler/mistral7b-ar-tokenizer-swap-pure-bf16-anneal-ablation",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use konstantindobler/mistral7b-ar-tokenizer-swap-pure-bf16-anneal-ablation with Docker Model Runner:
```
docker model run hf.co/konstantindobler/mistral7b-ar-tokenizer-swap-pure-bf16-anneal-ablation
```

mistral7b-ar-tokenizer-swap-pure-bf16-anneal-ablation

Mistral-7B-v0.1 adapted to Arabic as part of our study on efficient language adaptation: "Language Adaptation on a Tight Academic Compute Budget: Tokenizer Swapping Works and Pure bfloat16 Is Enough".

Code: https://github.com/konstantinjdobler/tight-budget-llm-adaptation

Paper: https://openreview.net/forum?id=VYfJaHeVod

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("konstantindobler/mistral7b-ar-tokenizer-swap-pure-bf16-anneal-ablation")
model = AutoModelForCausalLM.from_pretrained("konstantindobler/mistral7b-ar-tokenizer-swap-pure-bf16-anneal-ablation")

# Use model and tokenizer as usual

Details

The model is based on Mistral-7B-v0.1 and was adapted to Arabic. The original tokenizer was replaced by a language-specific Arabic tokenizer with a vocabulary of 32768 tokens. The new embeddings were initialized with FOCUS. Additionally, we tuned just the embeddings for 100 steps before training the full model. The model was then trained on 8 billion Arabic tokens from uonlp/CulturaX with pure bfloat16 precision (no mixed precision). However, in the final annealing phase of the learning rate schedule, the model was again trained using bfloat16 mixed precision. More details and hyperparameters can be found in the paper.

Disclaimer

The web-scale dataset used for pretraining and tokenizer training (uonlp/CulturaX) might contain personal and sensitive information. Such behavior needs to be assessed carefully before any real-world deployment of the models.

Citation

Please cite as follows:

@inproceedings{dobler2024language,
    title={Language Adaptation on a Tight Academic Compute Budget: Tokenizer Swapping Works and Pure bfloat16 Is Enough},
    author={Konstantin Dobler and Gerard de Melo},
    booktitle={2nd Workshop on Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization (WANT@ICML 2024)},
    year={2024},
    url={https://openreview.net/forum?id=VYfJaHeVod}
}

Acknowledgements

The project on which this model is based was funded by the Federal Ministry of Education and Research under the funding code "KI-Servicezentrum Berlin-Brandenburg" 01IS22092. Responsibility for the content of this publication remains with the author.

Downloads last month: 17

Safetensors

Model size

7B params

Tensor type

F32

konstantindobler
/

mistral7b-ar-tokenizer-swap-pure-bf16-anneal-ablation