Instructions to use eoinedge/edgeai-docs-qwen2.5-coder-0.5b-lora with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use eoinedge/edgeai-docs-qwen2.5-coder-0.5b-lora with PEFT:

from peft import PeftModel
from transformers import AutoModelForCausalLM

base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-Coder-0.5B-Instruct")
model = PeftModel.from_pretrained(base_model, "eoinedge/edgeai-docs-qwen2.5-coder-0.5b-lora")

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use eoinedge/edgeai-docs-qwen2.5-coder-0.5b-lora with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf eoinedge/edgeai-docs-qwen2.5-coder-0.5b-lora:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf eoinedge/edgeai-docs-qwen2.5-coder-0.5b-lora:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf eoinedge/edgeai-docs-qwen2.5-coder-0.5b-lora:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf eoinedge/edgeai-docs-qwen2.5-coder-0.5b-lora:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf eoinedge/edgeai-docs-qwen2.5-coder-0.5b-lora:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf eoinedge/edgeai-docs-qwen2.5-coder-0.5b-lora:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf eoinedge/edgeai-docs-qwen2.5-coder-0.5b-lora:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf eoinedge/edgeai-docs-qwen2.5-coder-0.5b-lora:Q4_K_M

Use Docker

docker model run hf.co/eoinedge/edgeai-docs-qwen2.5-coder-0.5b-lora:Q4_K_M

LM Studio
Jan

vLLM

How to use eoinedge/edgeai-docs-qwen2.5-coder-0.5b-lora with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "eoinedge/edgeai-docs-qwen2.5-coder-0.5b-lora"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "eoinedge/edgeai-docs-qwen2.5-coder-0.5b-lora",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/eoinedge/edgeai-docs-qwen2.5-coder-0.5b-lora:Q4_K_M

Ollama
How to use eoinedge/edgeai-docs-qwen2.5-coder-0.5b-lora with Ollama:
```
ollama run hf.co/eoinedge/edgeai-docs-qwen2.5-coder-0.5b-lora:Q4_K_M
```
Unsloth Desktop

How to use eoinedge/edgeai-docs-qwen2.5-coder-0.5b-lora with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf eoinedge/edgeai-docs-qwen2.5-coder-0.5b-lora:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "eoinedge/edgeai-docs-qwen2.5-coder-0.5b-lora:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use eoinedge/edgeai-docs-qwen2.5-coder-0.5b-lora with Docker Model Runner:
```
docker model run hf.co/eoinedge/edgeai-docs-qwen2.5-coder-0.5b-lora:Q4_K_M
```

Lemonade

How to use eoinedge/edgeai-docs-qwen2.5-coder-0.5b-lora with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull eoinedge/edgeai-docs-qwen2.5-coder-0.5b-lora:Q4_K_M

Run and chat with the model

lemonade run user.edgeai-docs-qwen2.5-coder-0.5b-lora-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use eoinedge/edgeai-docs-qwen2.5-coder-0.5b-lora with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf eoinedge/edgeai-docs-qwen2.5-coder-0.5b-lora:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default eoinedge/edgeai-docs-qwen2.5-coder-0.5b-lora:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use eoinedge/edgeai-docs-qwen2.5-coder-0.5b-lora with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf eoinedge/edgeai-docs-qwen2.5-coder-0.5b-lora:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "eoinedge/edgeai-docs-qwen2.5-coder-0.5b-lora:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

eoinedge commited on May 12

Commit

338bb27

verified ·

1 Parent(s): 0321606

Update model card: real training details, LoRA config, usage examples, related models

Browse files

Files changed (1) hide show

README.md +113 -135

README.md CHANGED Viewed

@@ -1,188 +1,166 @@
 ---
 base_model: Qwen/Qwen1.5-0.5B
 library_name: peft
-pipeline_tag: text-generation
-license: apache-2.0
 language:
-- en
 tags:
-- text-generation
-- conversational
-- lora
-- peft
-- transformers
-- safetensors
-- edge-deployment
-- base_model:adapter:Qwen/Qwen1.5-0.5B
 ---
 # edgeai-docs-embedding-qwen1.5-0.5b-instruct
-its still pretty terrible on performance but I hope to improve it with some training and am looking into an adaptor as an option too preview here:
-![image](https://cdn-uploads.huggingface.co/production/uploads/65fdb7d09a6eb46bd3d66d91/taGfx3w8SngcRABinHqRV.png)
-## Model Summary
-`edgeai-docs-embedding-qwen1.5-0.5b-instruct` is a lightweight, instruction-tuned LoRA adapter built on top of **Qwen1.5-0.5B**.
-It is optimized for:
-- Documentation summarization
-- Conversational responses
-- Lightweight embedding-style semantic generation
-- Edge and local deployment environments
-The model is designed to run efficiently in constrained environments and integrates cleanly with local inference stacks such as Ollama and llama.cpp.
 ---
-# Model Details
-## Model Description
-This model applies Parameter-Efficient Fine-Tuning (LoRA) to the base `Qwen/Qwen1.5-0.5B` model using custom instruction-formatted documentation datasets.
-It is tuned for structured reasoning over API documentation, developer content, and technical text. The objective is to provide high-quality summarization and conversational assistance while maintaining a small deployment footprint.
-### Developed by
-Eoin Jordan
-Hugging Face: https://huggingface.co/eoinedge
-### Model Type
-Instruction-tuned causal language model (LoRA adapter)
-### Base Model
-Qwen/Qwen1.5-0.5B
-### Fine-Tuning Method
-LoRA (Low-Rank Adaptation) via PEFT
-### Framework
-- Hugging Face Transformers
-- PEFT
-- Safetensors
-### Language
-English
-### License
-Apache 2.0
 ---
-# Intended Uses
-## Direct Use
-- Developer documentation summarization
-- API explanation generation
-- Conversational assistant for technical workflows
-- Lightweight semantic generation for search/retrieval pipelines
-## Downstream Use
-- Integrated into RAG (Retrieval-Augmented Generation) systems
-- Embedded inside edge devices
-- Wrapped within Ollama or llama.cpp inference environments
-- Developer tooling assistants
-## Out-of-Scope Use
-- High-stakes decision-making systems
-- Legal or medical advisory systems
-- Large-scale multi-lingual production deployments
-- Safety-critical automation
 ---
-# Bias, Risks, and Limitations
-Because this model is derived from Qwen1.5 and fine-tuned on technical documentation:
-- It may inherit biases present in the base model.
-- It is optimized for technical content and may degrade on general creative tasks.
-- It may hallucinate undocumented APIs or incorrect implementation details.
-- It is not designed for factual verification tasks.
-Users should validate outputs before use in production systems.
----
-# Training Details
-## Training Data
-- Custom documentation chunks
-- API references
-- Instruction-formatted JSON datasets
-- Manually validated prompt-response pairs
-No personally identifiable information (PII) was intentionally included.
-## Training Procedure
-- LoRA fine-tuning via PEFT
-- Instruction tuning objective
-- Mixed precision training
-### Training Regime
-bf16 mixed precision
----
-# Evaluation
-Evaluation was performed using:
-- Manual prompt benchmarking
-- Structured documentation summarization tests
-- API explanation accuracy review
-Quantitative benchmarks are limited due to domain-specific fine-tuning focus.
 ---
-# Technical Specifications
-## Architecture
-- Transformer decoder-only model
-- 0.5B parameters (base model)
-- LoRA adapter layers applied to attention modules
-## Adapter Type
-Low-Rank Adaptation (LoRA)
-## Deployment Targets
-- Ollama
-- llama.cpp (via conversion)
-- Transformers runtime
-- Edge inference environments
 ---
-# Environmental Impact
-Base model reference:
-Qwen Technical Report (2023)
-Carbon emissions were not separately tracked for this adapter training run.
-For estimation guidance, see: https://mlco2.github.io/impact
 ---
-# Citation
-If you use this model, please cite:
 ```bibtex
 @misc{edgeai-docs-embedding-qwen1.5-0.5b-instruct,
-  author = {Jordan, Eoin},
-  title = {edgeai-docs-embedding-qwen1.5-0.5b-instruct},
-  year = {2026},
   howpublished = {\url{https://huggingface.co/eoinedge/edgeai-docs-embedding-qwen1.5-0.5b-instruct}}
-}

 ---
+license: apache-2.0
 base_model: Qwen/Qwen1.5-0.5B
 library_name: peft
 language:
+  - en
 tags:
+  - lora
+  - peft
+  - qwen
+  - edge-ai
+  - edge-impulse
+  - documentation
+  - code-generation
+  - conversational
+pipeline_tag: text-generation
 ---
 # edgeai-docs-embedding-qwen1.5-0.5b-instruct
+A LoRA adapter fine-tuned on **1,794 Edge Impulse / Edge AI MDX documentation files** from the [Edge Impulse documentation](https://docs.edgeimpulse.com), built on top of [`Qwen/Qwen1.5-0.5B`](https://huggingface.co/Qwen/Qwen1.5-0.5B).
+Optimized for:
+- Answering developer questions about Edge Impulse Studio, SDK, and APIs
+- Summarizing technical documentation
+- Generating code snippets for edge ML workflows
+- Lightweight local/edge deployment
+> **Larger variants in training:** [1.5B](https://huggingface.co/eoinedge/edgeai-qwen2.5coder-1.5b-lora) · [7B](https://huggingface.co/eoinedge/edgeai-qwen2.5coder-7b-lora) (Qwen2.5-Coder base)
 ---
+## Model Details
+| Property | Value |
+|---|---|
+| Base model | `Qwen/Qwen1.5-0.5B` |
+| Adapter type | LoRA (PEFT) |
+| LoRA rank (`r`) | 8 |
+| LoRA alpha | 32 |
+| Target modules | `q_proj`, `v_proj` |
+| LoRA dropout | 0.05 |
+| Trainable parameters | ~786K (0.17% of base) |
+| Training epochs | 3 |
+| Batch size | 4 (× grad accum 2 = effective 8) |
+| Learning rate | 3e-4 (cosine decay) |
+| Sequence length | 512 tokens |
+| Training hardware | Apple M1 Pro (MPS, fp16) |
+| Precision | float16 |
 ---
+## Training Data
+| Stat | Value |
+|---|---|
+| Source | [Edge Impulse Mintlify documentation](https://docs.edgeimpulse.com) |
+| File format | MDX (Markdown + JSX components) |
+| Total files | 1,794 `.mdx` files |
+| Preprocessing | Frontmatter, JSX tags, imports stripped; code fences unwrapped; links flattened |
+| Chunks generated | ~3,500 × 512-token chunks |
+Topics covered: Studio projects, datasets, DSP blocks, learning blocks, deployment targets, Python SDK, REST API, CLI tools, edge inference, model optimization, and more.
 ---
+## Usage
+### Load and chat
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+from peft import PeftModel
+BASE_MODEL = "Qwen/Qwen1.5-0.5B"
+ADAPTER    = "eoinedge/edgeai-docs-embedding-qwen1.5-0.5b-instruct"
+device = "cuda" if torch.cuda.is_available() else \
+         "mps"  if torch.backends.mps.is_available() else "cpu"
+tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)
+base  = AutoModelForCausalLM.from_pretrained(BASE_MODEL,
+            dtype=torch.float16 if device != "cpu" else torch.float32,
+            device_map=device)
+model = PeftModel.from_pretrained(base, ADAPTER)
+model.eval()
+```
+### Single question
+```python
+prompt = "How do I collect sensor data using the Edge Impulse data forwarder?"
+inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+with torch.no_grad():
+    out = model.generate(**inputs, max_new_tokens=512,
+                         do_sample=True, temperature=0.7, top_p=0.95,
+                         pad_token_id=tokenizer.eos_token_id)
+print(tokenizer.decode(out[0], skip_special_tokens=True))
+```
+### Chat template (multi-turn)
+```python
+messages = [
+    {"role": "user", "content": "What is an impulse in Edge Impulse?"}
+]
+formatted = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+inputs = tokenizer(formatted, return_tensors="pt").to(model.device)
+with torch.no_grad():
+    out = model.generate(**inputs, max_new_tokens=512,
+                         do_sample=True, temperature=0.7,
+                         pad_token_id=tokenizer.eos_token_id)
+print(tokenizer.decode(out[0], skip_special_tokens=True))
+```
 ---
+## Example Prompts
+| Task | Prompt |
+|---|---|
+| Concept explanation | `"What is a DSP block in Edge Impulse?"` |
+| API usage | `"How do I use the Edge Impulse Python SDK to upload data?"` |
+| Deployment | `"How do I deploy a model to an Arduino Nano 33 BLE Sense?"` |
+| Code generation | `"Write Python code to collect IMU data and upload it to Edge Impulse."` |
+| Troubleshooting | `"Why is my Edge Impulse model showing high latency on the Cortex-M4?"` |
+---
+## Limitations
+- Based on a 0.5B parameter base model — may struggle with complex multi-step reasoning
+- Training data covers documentation as of mid-2026; newer APIs may not be represented
+- May hallucinate undocumented Edge Impulse features
+- Not suitable for safety-critical or production decision-making systems
+- Validate generated code before running on real hardware
 ---
+## Related Models
+| Model | Base | Status |
+|---|---|---|
+| This model | Qwen1.5-0.5B | ✅ Available |
+| [eoinedge/edgeai-qwen2.5coder-1.5b-lora](https://huggingface.co/eoinedge/edgeai-qwen2.5coder-1.5b-lora) | Qwen2.5-Coder-1.5B-Instruct | 🔄 Training |
+| [eoinedge/edgeai-qwen2.5coder-7b-lora](https://huggingface.co/eoinedge/edgeai-qwen2.5coder-7b-lora) | Qwen2.5-Coder-7B-Instruct | 🔄 Training |
+| [eoinedge/arduino-qwen0.5-lora](https://huggingface.co/eoinedge/arduino-qwen0.5-lora) | Qwen1.5-0.5B | ✅ Available (Arduino docs) |
 ---
+## Citation
 ```bibtex
 @misc{edgeai-docs-embedding-qwen1.5-0.5b-instruct,
+  author       = {Jordan, Eoin},
+  title        = {edgeai-docs-embedding-qwen1.5-0.5b-instruct},
+  year         = {2026},
+  publisher    = {Hugging Face},
   howpublished = {\url{https://huggingface.co/eoinedge/edgeai-docs-embedding-qwen1.5-0.5b-instruct}}
+}
+```