Instructions to use katanemo/Arch-Function-3B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use katanemo/Arch-Function-3B with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="katanemo/Arch-Function-3B") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("katanemo/Arch-Function-3B") model = AutoModelForCausalLM.from_pretrained("katanemo/Arch-Function-3B", device_map="auto") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=40) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use katanemo/Arch-Function-3B with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "katanemo/Arch-Function-3B" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "katanemo/Arch-Function-3B", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/katanemo/Arch-Function-3B
- SGLang
How to use katanemo/Arch-Function-3B with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "katanemo/Arch-Function-3B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "katanemo/Arch-Function-3B", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "katanemo/Arch-Function-3B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "katanemo/Arch-Function-3B", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use katanemo/Arch-Function-3B with Docker Model Runner:
docker model run hf.co/katanemo/Arch-Function-3B
| # Katanemo Labs, Inc. COMMUNITY LICENSE AGREEMENT | |
| **Version Release Date:** September 30th, 2024 | |
| This Katanemo Labs, Inc. COMMUNITY LICENSE AGREEMENT is based on the Llama 3.2 Community License, Copyright © Meta Platforms, Inc. The terms and conditions have been adapted to reflect the proprietary nature of Katanemo Labs' materials. | |
| 1.Definitions | |
| a. "Agreement": The terms and conditions for use, reproduction, distribution, and modification of the Katanemo Materials set forth herein. | |
| b. "Documentation": The specifications, manuals, and documentation accompanying Katanemo LLMs v1. | |
| c. "Licensee" or "you: The individual or entity entering into this Agreement, including your employer if you are acting on their behalf. | |
| d. "Katanemo": The foundational large language models and software provided by Katanemo Labs, Inc., available at https://huggingface.co/katanemolabs. | |
| e. "Katanemo Materials": Collectively, Katanemo's proprietary models and Documentation. Some Materials are derived from the Qwen language models licensed under the Qwen RESEARCH LICENSE AGREEMENT. | |
| f. "Katanemo Labs" or "we": Katanemo Labs Inc., a Delaware, USA Corporation. | |
| 2. | |
| By clicking "I Accept" or using any part of the Katanemo Materials, you agree to be bound by this Agreement. | |
| 3. License Rights and Redistribution | |
| a. Grant of Rights | |
| You are granted a non-exclusive, worldwide, non-transferable, and royalty-free license to: | |
| - Use, reproduce, distribute, and modify the Katanemo Materials. | |
| - Create derivative works based on the Katanemo Materials. | |
| 4. Redistribution and Use | |
| a. Distribution: | |
| If you distribute the Katanemo Materials or a derivative work: | |
| - Include a copy of this Agreement. | |
| - Prominently display "Built with Katanemo" on a related website or documentation. | |
| b. Attribution: | |
| Include the following attribution notice: | |
| "Katanemo is licensed under the Katanemo Labs Community License, Copyright © Katanemo Labs, Inc. All Rights Reserved."_ | |
| c. Compliance: | |
| Your use must adhere to the Acceptable Use Policy, available at https://katanemolabs.com/katanemo/use-policy. | |
| 5. Additional Commercial Terms | |
| If you are commercially using the Materials, you shall request a license from us. | |
| 6. Disclaimer of Warranty | |
| The Katanemo Materials are provided "AS IS" without warranties of any kind, either express or implied, including but not limited to warranties of title, non-infringement, or fitness for a particular purpose. | |
| 7. Limitation of Liability | |
| Katanemo Labs is not liable for any indirect, special, or consequential damages arising out of the use of the Katanemo Materials, even if advised of the possibility of such damages. | |
| 8. Intellectual Property | |
| a. Trademarks | |
| No trademark licenses are granted, except as required for attribution as described in Section 1.b. You may use the “Katanemo” mark according to Katanemo Labs' brand guidelines. | |
| b. Ownership | |
| You own any derivative works or modifications you create, except for portions owned by Katanemo Labs. | |
| c. Litigation | |
| If you file a lawsuit against Katanemo Labs regarding intellectual property, your license under this Agreement terminates. | |
| 9. Term and Termination | |
| This Agreement continues until terminated. Katanemo Labs may terminate the Agreement if you breach any terms. Upon termination, you must cease using the Katanemo Materials. | |
| 10. Governing Law and Jurisdiction | |
| This Agreement is governed by the laws of the State of Washington, USA. Any disputes will be resolved in the courts of California. | |