Instructions to use mradermacher/Qwen3-Embedding-Medical-0.6B-i1-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use mradermacher/Qwen3-Embedding-Medical-0.6B-i1-GGUF with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("feature-extraction", model="mradermacher/Qwen3-Embedding-Medical-0.6B-i1-GGUF")

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("mradermacher/Qwen3-Embedding-Medical-0.6B-i1-GGUF", device_map="auto")

sentence-transformers

How to use mradermacher/Qwen3-Embedding-Medical-0.6B-i1-GGUF with sentence-transformers:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("mradermacher/Qwen3-Embedding-Medical-0.6B-i1-GGUF")

sentences = [
    "The weather is lovely today.",
    "It's so sunny outside!",
    "He drove to the stadium."
]
embeddings = model.encode(sentences)

similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use mradermacher/Qwen3-Embedding-Medical-0.6B-i1-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf mradermacher/Qwen3-Embedding-Medical-0.6B-i1-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf mradermacher/Qwen3-Embedding-Medical-0.6B-i1-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf mradermacher/Qwen3-Embedding-Medical-0.6B-i1-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf mradermacher/Qwen3-Embedding-Medical-0.6B-i1-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf mradermacher/Qwen3-Embedding-Medical-0.6B-i1-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf mradermacher/Qwen3-Embedding-Medical-0.6B-i1-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf mradermacher/Qwen3-Embedding-Medical-0.6B-i1-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf mradermacher/Qwen3-Embedding-Medical-0.6B-i1-GGUF:Q4_K_M

Use Docker

docker model run hf.co/mradermacher/Qwen3-Embedding-Medical-0.6B-i1-GGUF:Q4_K_M

LM Studio
Jan
Ollama
How to use mradermacher/Qwen3-Embedding-Medical-0.6B-i1-GGUF with Ollama:
```
ollama run hf.co/mradermacher/Qwen3-Embedding-Medical-0.6B-i1-GGUF:Q4_K_M
```
Unsloth Desktop

How to use mradermacher/Qwen3-Embedding-Medical-0.6B-i1-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf mradermacher/Qwen3-Embedding-Medical-0.6B-i1-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "mradermacher/Qwen3-Embedding-Medical-0.6B-i1-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use mradermacher/Qwen3-Embedding-Medical-0.6B-i1-GGUF with Docker Model Runner:
```
docker model run hf.co/mradermacher/Qwen3-Embedding-Medical-0.6B-i1-GGUF:Q4_K_M
```

Lemonade

How to use mradermacher/Qwen3-Embedding-Medical-0.6B-i1-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull mradermacher/Qwen3-Embedding-Medical-0.6B-i1-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Qwen3-Embedding-Medical-0.6B-i1-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use mradermacher/Qwen3-Embedding-Medical-0.6B-i1-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf mradermacher/Qwen3-Embedding-Medical-0.6B-i1-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default mradermacher/Qwen3-Embedding-Medical-0.6B-i1-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use mradermacher/Qwen3-Embedding-Medical-0.6B-i1-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf mradermacher/Qwen3-Embedding-Medical-0.6B-i1-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "mradermacher/Qwen3-Embedding-Medical-0.6B-i1-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

mradermacher commited on Aug 13

Commit

c1bc394

verified ·

1 Parent(s): 40ac203

uploaded from rich1

Browse files

Files changed (25) hide show

.gitattributes +24 -0
Qwen3-Embedding-Medical-0.6B.i1-IQ1_M.gguf +3 -0
Qwen3-Embedding-Medical-0.6B.i1-IQ1_S.gguf +3 -0
Qwen3-Embedding-Medical-0.6B.i1-IQ2_M.gguf +3 -0
Qwen3-Embedding-Medical-0.6B.i1-IQ2_S.gguf +3 -0
Qwen3-Embedding-Medical-0.6B.i1-IQ2_XS.gguf +3 -0
Qwen3-Embedding-Medical-0.6B.i1-IQ2_XXS.gguf +3 -0
Qwen3-Embedding-Medical-0.6B.i1-IQ3_M.gguf +3 -0
Qwen3-Embedding-Medical-0.6B.i1-IQ3_S.gguf +3 -0
Qwen3-Embedding-Medical-0.6B.i1-IQ3_XS.gguf +3 -0
Qwen3-Embedding-Medical-0.6B.i1-IQ3_XXS.gguf +3 -0
Qwen3-Embedding-Medical-0.6B.i1-IQ4_NL.gguf +3 -0
Qwen3-Embedding-Medical-0.6B.i1-IQ4_XS.gguf +3 -0
Qwen3-Embedding-Medical-0.6B.i1-Q2_K.gguf +3 -0
Qwen3-Embedding-Medical-0.6B.i1-Q2_K_S.gguf +3 -0
Qwen3-Embedding-Medical-0.6B.i1-Q3_K_L.gguf +3 -0
Qwen3-Embedding-Medical-0.6B.i1-Q3_K_M.gguf +3 -0
Qwen3-Embedding-Medical-0.6B.i1-Q3_K_S.gguf +3 -0
Qwen3-Embedding-Medical-0.6B.i1-Q4_0.gguf +3 -0
Qwen3-Embedding-Medical-0.6B.i1-Q4_1.gguf +3 -0
Qwen3-Embedding-Medical-0.6B.i1-Q4_K_M.gguf +3 -0
Qwen3-Embedding-Medical-0.6B.i1-Q4_K_S.gguf +3 -0
Qwen3-Embedding-Medical-0.6B.i1-Q5_K_M.gguf +3 -0
Qwen3-Embedding-Medical-0.6B.i1-Q5_K_S.gguf +3 -0
Qwen3-Embedding-Medical-0.6B.i1-Q6_K.gguf +3 -0

.gitattributes CHANGED Viewed

@@ -34,3 +34,27 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 Qwen3-Embedding-Medical-0.6B.imatrix.gguf filter=lfs diff=lfs merge=lfs -text

 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 Qwen3-Embedding-Medical-0.6B.imatrix.gguf filter=lfs diff=lfs merge=lfs -text
+Qwen3-Embedding-Medical-0.6B.i1-IQ1_M.gguf filter=lfs diff=lfs merge=lfs -text
+Qwen3-Embedding-Medical-0.6B.i1-IQ1_S.gguf filter=lfs diff=lfs merge=lfs -text
+Qwen3-Embedding-Medical-0.6B.i1-IQ2_M.gguf filter=lfs diff=lfs merge=lfs -text
+Qwen3-Embedding-Medical-0.6B.i1-IQ2_S.gguf filter=lfs diff=lfs merge=lfs -text
+Qwen3-Embedding-Medical-0.6B.i1-IQ2_XS.gguf filter=lfs diff=lfs merge=lfs -text
+Qwen3-Embedding-Medical-0.6B.i1-IQ2_XXS.gguf filter=lfs diff=lfs merge=lfs -text
+Qwen3-Embedding-Medical-0.6B.i1-IQ3_M.gguf filter=lfs diff=lfs merge=lfs -text
+Qwen3-Embedding-Medical-0.6B.i1-IQ3_S.gguf filter=lfs diff=lfs merge=lfs -text
+Qwen3-Embedding-Medical-0.6B.i1-IQ3_XS.gguf filter=lfs diff=lfs merge=lfs -text
+Qwen3-Embedding-Medical-0.6B.i1-IQ3_XXS.gguf filter=lfs diff=lfs merge=lfs -text
+Qwen3-Embedding-Medical-0.6B.i1-IQ4_NL.gguf filter=lfs diff=lfs merge=lfs -text
+Qwen3-Embedding-Medical-0.6B.i1-IQ4_XS.gguf filter=lfs diff=lfs merge=lfs -text
+Qwen3-Embedding-Medical-0.6B.i1-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
+Qwen3-Embedding-Medical-0.6B.i1-Q2_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+Qwen3-Embedding-Medical-0.6B.i1-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
+Qwen3-Embedding-Medical-0.6B.i1-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+Qwen3-Embedding-Medical-0.6B.i1-Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+Qwen3-Embedding-Medical-0.6B.i1-Q4_0.gguf filter=lfs diff=lfs merge=lfs -text
+Qwen3-Embedding-Medical-0.6B.i1-Q4_1.gguf filter=lfs diff=lfs merge=lfs -text
+Qwen3-Embedding-Medical-0.6B.i1-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+Qwen3-Embedding-Medical-0.6B.i1-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+Qwen3-Embedding-Medical-0.6B.i1-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+Qwen3-Embedding-Medical-0.6B.i1-Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+Qwen3-Embedding-Medical-0.6B.i1-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text

Qwen3-Embedding-Medical-0.6B.i1-IQ1_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:84d992b28a5ddbe4de1dca773dcdaee03e1354ad7fa6a7a35f36e4e4968ace85
+size 215858976

Qwen3-Embedding-Medical-0.6B.i1-IQ1_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:be440d60740575a4f53ff9acf4b1fee4734ce1579a57d2ef238b9cab4f5b20c8
+size 207822624

Qwen3-Embedding-Medical-0.6B.i1-IQ2_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:52220e053106082d536d8ba14a5d6ccc53fd71afd144a04803a14d0f59552a92
+size 264716064

Qwen3-Embedding-Medical-0.6B.i1-IQ2_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b035107853b4f1033e477f7ae7da149afdab0df93ff4c8a80f5dd170b5fd2089
+size 254000928

Qwen3-Embedding-Medical-0.6B.i1-IQ2_XS.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c2216847d648a5ad68ec86cc94de82870f783219b27427ec0007b918df58809f
+size 241803040

Qwen3-Embedding-Medical-0.6B.i1-IQ2_XXS.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:312291026a5b648402e288296b65171be4a45d2f293219c442b4d1d484e47282
+size 229252896

Qwen3-Embedding-Medical-0.6B.i1-IQ3_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:d5aafe4e7b63d507f3ed3c17bd5022a4d067e5ddba32f946d0ecbe883c7dffd8
+size 335797600

Qwen3-Embedding-Medical-0.6B.i1-IQ3_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:48b533be47d762b6daf7f3a92f48286adb70ccc7f7243889df135c36fd61dbb6
+size 322846048

Qwen3-Embedding-Medical-0.6B.i1-IQ3_XS.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:56ed35b0848c8dfbbbad89c630afefc279b4b260d095ba7dc12e20cad57aa69c
+size 312524128

Qwen3-Embedding-Medical-0.6B.i1-IQ3_XXS.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a82ad425464842cfe7d7c1adc855c23c17ff26000ecb999ba2371f0dbf7e5e9b
+size 278822688

Qwen3-Embedding-Medical-0.6B.i1-IQ4_NL.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:dbdca7c556928985922ac10547b84755045068c92a0f84d4ff6b1d5605dcfd47
+size 381336928

Qwen3-Embedding-Medical-0.6B.i1-IQ4_XS.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:14f63c1bbb151ce98aa643ae77b1da8f4f49c3d6846e65ee339a2227ad13c6f7
+size 367574368

Qwen3-Embedding-Medical-0.6B.i1-Q2_K.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:7791ed2323097ffad43db3f5b5fdd56442b4cab74d1e89b85497e910b13fee38
+size 296009056

Qwen3-Embedding-Medical-0.6B.i1-Q2_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:92d671a8a8f26d4375f6bb6590f259a9143e0e6794505dd4fb8e16890a678eac
+size 280329568

Qwen3-Embedding-Medical-0.6B.i1-Q3_K_L.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:078aeb22e55e968bc3f703301b866c4cad03a9d12f9b9884b19fb9d222144c8a
+size 368262496

Qwen3-Embedding-Medical-0.6B.i1-Q3_K_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c8c4edbd064090bbfaa54364636f5181da6c78eede84a3a7d27535aef905bd68
+size 346897760

Qwen3-Embedding-Medical-0.6B.i1-Q3_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c6dc1da74ffc6c2aa33904de8c5d6c40e57471c952606fe1b5e4ed05baf252fb
+size 322846048

Qwen3-Embedding-Medical-0.6B.i1-Q4_0.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:bade7af83e635d14677b891a300cbadde6b67e9c60d3fe5285f769b031118088
+size 381926752

Qwen3-Embedding-Medical-0.6B.i1-Q4_1.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f109bc84b6365727038a5ca098c40457b8f60ebc0c74475668ab6025044b51d1
+size 408862048

Qwen3-Embedding-Medical-0.6B.i1-Q4_K_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:0ecbe65d6d2798374ae1efb46cb399cfb0a533739cbb690ef3552101e8fec1e0
+size 396475744

Qwen3-Embedding-Medical-0.6B.i1-Q4_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:568df3e4503ec220d8f7c1b70a3d7ad4a8c44af80a56cc8db4d234178825b422
+size 383040864

Qwen3-Embedding-Medical-0.6B.i1-Q5_K_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f48253419208a33c3e295e575d2d381634139e5a34b6a2920ed3a4f007217342
+size 444185952

Qwen3-Embedding-Medical-0.6B.i1-Q5_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:7d69d4d60ce42fe2ee181ecadc4627fef95658cac0ef4fa2b220ad5a76ad1fed
+size 436387168

Qwen3-Embedding-Medical-0.6B.i1-Q6_K.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:2b009e2ddcc9d3683782d64c63c98dab5795a3641380d0076d2849ca2b918635
+size 494878048