Instructions to use willtensora/fd1980a0-7e71-4e52-addb-318dca5991d5 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use willtensora/fd1980a0-7e71-4e52-addb-318dca5991d5 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="willtensora/fd1980a0-7e71-4e52-addb-318dca5991d5")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("willtensora/fd1980a0-7e71-4e52-addb-318dca5991d5")
model = AutoModelForCausalLM.from_pretrained("willtensora/fd1980a0-7e71-4e52-addb-318dca5991d5", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use willtensora/fd1980a0-7e71-4e52-addb-318dca5991d5 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "willtensora/fd1980a0-7e71-4e52-addb-318dca5991d5"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "willtensora/fd1980a0-7e71-4e52-addb-318dca5991d5",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/willtensora/fd1980a0-7e71-4e52-addb-318dca5991d5

SGLang

How to use willtensora/fd1980a0-7e71-4e52-addb-318dca5991d5 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "willtensora/fd1980a0-7e71-4e52-addb-318dca5991d5" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "willtensora/fd1980a0-7e71-4e52-addb-318dca5991d5",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "willtensora/fd1980a0-7e71-4e52-addb-318dca5991d5" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "willtensora/fd1980a0-7e71-4e52-addb-318dca5991d5",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use willtensora/fd1980a0-7e71-4e52-addb-318dca5991d5 with Docker Model Runner:
```
docker model run hf.co/willtensora/fd1980a0-7e71-4e52-addb-318dca5991d5
```

See axolotl config

axolotl version: 0.4.1

base_model: fxmarty/tiny-random-GemmaForCausalLM
batch_size: 32
bf16: true
chat_template: tokenizer_default_fallback_alpaca
datasets:
- data_files:
  - b7c2a4a781c93416_train_data.json
  ds_type: json
  format: custom
  path: /workspace/input_data/b7c2a4a781c93416_train_data.json
  type:
    field_input: context
    field_instruction: question
    field_output: answer
    format: '{instruction} {input}'
    no_input_format: '{instruction}'
    system_format: '{system}'
    system_prompt: ''
eval_steps: 20
flash_attention: true
gpu_memory_limit: 80GiB
gradient_checkpointing: true
group_by_length: true
hub_model_id: willtensora/fd1980a0-7e71-4e52-addb-318dca5991d5
hub_strategy: checkpoint
learning_rate: 0.0002
logging_steps: 10
lr_scheduler: cosine
max_steps: 2500
micro_batch_size: 4
model_type: AutoModelForCausalLM
optimizer: adamw_bnb_8bit
output_dir: /workspace/axolotl/configs
pad_to_sequence_len: true
resize_token_embeddings_to_32x: false
sample_packing: false
save_steps: 40
save_total_limit: 1
sequence_len: 2048
tokenizer_type: GemmaTokenizerFast
train_on_inputs: false
trust_remote_code: true
val_set_size: 0.1
wandb_entity: ''
wandb_mode: online
wandb_name: fxmarty/tiny-random-GemmaForCausalLM-/workspace/input_data/b7c2a4a781c93416_train_data.json
wandb_project: Gradients-On-Demand
wandb_run: your_name
wandb_runid: default
warmup_ratio: 0.05
xformers_attention: true

fd1980a0-7e71-4e52-addb-318dca5991d5

This model is a fine-tuned version of fxmarty/tiny-random-GemmaForCausalLM on the None dataset. It achieves the following results on the evaluation set:

Loss: 11.7971

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 0.0002
train_batch_size: 4
eval_batch_size: 4
seed: 42
distributed_type: multi-GPU
num_devices: 8
total_train_batch_size: 32
total_eval_batch_size: 32
optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: cosine
lr_scheduler_warmup_steps: 7
training_steps: 156

Training results

Training Loss	Epoch	Step	Validation Loss
No log	0.0008	1	12.4537
12.4357	0.0161	20	12.4267
12.392	0.0322	40	12.3762
12.3026	0.0483	60	12.2651
12.1177	0.0645	80	12.0658
11.9286	0.0806	100	11.8860
11.8324	0.0967	120	11.8100
11.798	0.1128	140	11.7971

Framework versions

Transformers 4.46.0
Pytorch 2.5.0+cu124
Datasets 3.0.1
Tokenizers 0.20.1

Downloads last month: 7

Safetensors

Model size

8.19M params

Tensor type

BF16

Model tree for willtensora/fd1980a0-7e71-4e52-addb-318dca5991d5

Base model

fxmarty/tiny-random-GemmaForCausalLM

Finetuned

(1)

this model