Instructions to use nayohan/llama3-8b-it-translation-tech-en-ko-1sent with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use nayohan/llama3-8b-it-translation-tech-en-ko-1sent with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="nayohan/llama3-8b-it-translation-tech-en-ko-1sent")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("nayohan/llama3-8b-it-translation-tech-en-ko-1sent")
model = AutoModelForCausalLM.from_pretrained("nayohan/llama3-8b-it-translation-tech-en-ko-1sent", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use nayohan/llama3-8b-it-translation-tech-en-ko-1sent with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "nayohan/llama3-8b-it-translation-tech-en-ko-1sent"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nayohan/llama3-8b-it-translation-tech-en-ko-1sent",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/nayohan/llama3-8b-it-translation-tech-en-ko-1sent

SGLang

How to use nayohan/llama3-8b-it-translation-tech-en-ko-1sent with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "nayohan/llama3-8b-it-translation-tech-en-ko-1sent" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nayohan/llama3-8b-it-translation-tech-en-ko-1sent",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "nayohan/llama3-8b-it-translation-tech-en-ko-1sent" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nayohan/llama3-8b-it-translation-tech-en-ko-1sent",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use nayohan/llama3-8b-it-translation-tech-en-ko-1sent with Docker Model Runner:
```
docker model run hf.co/nayohan/llama3-8b-it-translation-tech-en-ko-1sent
```

Introduction

The model was trained to translate a single sentence from English to Korean with a 1.3M dataset in the technology science domain. Dataset: nayohan/tech_science_translation

Loading the Model

Use the following Python code to load the model:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "nayohan/llama3-8b-translation-en-ko-1sent"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
  model_name,
  device_map="auto",
  torch_dtype=torch.bfloat16
)

Generating Text

To generate text, use the following Python code: No support for other languages or reverse direction and styles at this time.

source="en"
target="ko"
style="written"
SYSTEM_PROMPT=f"Acts as a translator. Translate {source} sentences into {target} sentences in {style} style."
s = "The aerospace industry is a flower in the field of technology and science."
conversation = [{'role': 'system', 'content': SYSTEM_PROMPT},
                {'role': 'user', 'content': s}]

inputs = tokenizer.apply_chat_template(
  conversation,
  tokenize=True,
  add_generation_prompt=True,
  return_tensors='pt'
).to("cuda")

outputs = model.generate(inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][len(inputs[0]):]))

# Result
# INPUT: <|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nActs as a translator. Translate en sentences into ko sentences in written style.<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nThe aerospace industry is a flower in the field of technology and science.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n
# OUTPUT: 항공 우주 산업은 기술과 과학 분야의 꽃이라고 할 수 있다.

## [Warning!] In multiple sentences, there is a tendency to output in a single sentence.
# INPUT: <|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nActs as a translator. Translate ko sentences into en sentences in written style.<|eot_id|><|start_header_id|>user<|end_header_id|>\n\n
Technical and basic sciences are very important in terms of research. It has a significant impact on the industrial development of a country. Government policies control the research budget.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n
# OUTPUT: 연구 측면에서 기술 및 기초 과학은 국가의 산업 발전에 큰 영향을 미치며 정부 정책은 연구 예산을 통제한다.

Citation

@article{llama3modelcard,
        title={Llama 3 Model Card},
        author={AI@Meta},
        year={2024},
        url={https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md}
}

Our trainig code can be found here: [TBD]

Downloads last month: 10

Safetensors

Model size

8B params

Tensor type

BF16

Model tree for nayohan/llama3-8b-it-translation-tech-en-ko-1sent

Base model

meta-llama/Meta-Llama-3-8B-Instruct

Finetuned

(1087)

this model

nayohan
/

llama3-8b-it-translation-tech-en-ko-1sent

Introduction

Loading the Model

Generating Text

Citation

Model tree for nayohan/llama3-8b-it-translation-tech-en-ko-1sent

Dataset used to train nayohan/llama3-8b-it-translation-tech-en-ko-1sent