Instructions to use mradermacher/OCRFlux-3B-i1-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use mradermacher/OCRFlux-3B-i1-GGUF with Transformers:

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("mradermacher/OCRFlux-3B-i1-GGUF", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use mradermacher/OCRFlux-3B-i1-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf mradermacher/OCRFlux-3B-i1-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf mradermacher/OCRFlux-3B-i1-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf mradermacher/OCRFlux-3B-i1-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf mradermacher/OCRFlux-3B-i1-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf mradermacher/OCRFlux-3B-i1-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf mradermacher/OCRFlux-3B-i1-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf mradermacher/OCRFlux-3B-i1-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf mradermacher/OCRFlux-3B-i1-GGUF:Q4_K_M

Use Docker

docker model run hf.co/mradermacher/OCRFlux-3B-i1-GGUF:Q4_K_M

LM Studio
Jan
Ollama
How to use mradermacher/OCRFlux-3B-i1-GGUF with Ollama:
```
ollama run hf.co/mradermacher/OCRFlux-3B-i1-GGUF:Q4_K_M
```
Unsloth Desktop

How to use mradermacher/OCRFlux-3B-i1-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf mradermacher/OCRFlux-3B-i1-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "mradermacher/OCRFlux-3B-i1-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use mradermacher/OCRFlux-3B-i1-GGUF with Docker Model Runner:
```
docker model run hf.co/mradermacher/OCRFlux-3B-i1-GGUF:Q4_K_M
```

Lemonade

How to use mradermacher/OCRFlux-3B-i1-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull mradermacher/OCRFlux-3B-i1-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.OCRFlux-3B-i1-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use mradermacher/OCRFlux-3B-i1-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf mradermacher/OCRFlux-3B-i1-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default mradermacher/OCRFlux-3B-i1-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use mradermacher/OCRFlux-3B-i1-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf mradermacher/OCRFlux-3B-i1-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "mradermacher/OCRFlux-3B-i1-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

mradermacher commited on Jun 28, 2025

Commit

97005fc

verified ·

1 Parent(s): e4ec7fc

uploaded from nico1

Browse files

Files changed (25) hide show

.gitattributes +24 -0
OCRFlux-3B.i1-IQ1_M.gguf +3 -0
OCRFlux-3B.i1-IQ1_S.gguf +3 -0
OCRFlux-3B.i1-IQ2_M.gguf +3 -0
OCRFlux-3B.i1-IQ2_S.gguf +3 -0
OCRFlux-3B.i1-IQ2_XS.gguf +3 -0
OCRFlux-3B.i1-IQ2_XXS.gguf +3 -0
OCRFlux-3B.i1-IQ3_M.gguf +3 -0
OCRFlux-3B.i1-IQ3_S.gguf +3 -0
OCRFlux-3B.i1-IQ3_XS.gguf +3 -0
OCRFlux-3B.i1-IQ3_XXS.gguf +3 -0
OCRFlux-3B.i1-IQ4_NL.gguf +3 -0
OCRFlux-3B.i1-IQ4_XS.gguf +3 -0
OCRFlux-3B.i1-Q2_K.gguf +3 -0
OCRFlux-3B.i1-Q2_K_S.gguf +3 -0
OCRFlux-3B.i1-Q3_K_L.gguf +3 -0
OCRFlux-3B.i1-Q3_K_M.gguf +3 -0
OCRFlux-3B.i1-Q3_K_S.gguf +3 -0
OCRFlux-3B.i1-Q4_0.gguf +3 -0
OCRFlux-3B.i1-Q4_1.gguf +3 -0
OCRFlux-3B.i1-Q4_K_M.gguf +3 -0
OCRFlux-3B.i1-Q4_K_S.gguf +3 -0
OCRFlux-3B.i1-Q5_K_M.gguf +3 -0
OCRFlux-3B.i1-Q5_K_S.gguf +3 -0
OCRFlux-3B.i1-Q6_K.gguf +3 -0

.gitattributes CHANGED Viewed

@@ -34,3 +34,27 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 imatrix.dat filter=lfs diff=lfs merge=lfs -text

 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 imatrix.dat filter=lfs diff=lfs merge=lfs -text
+OCRFlux-3B.i1-IQ1_M.gguf filter=lfs diff=lfs merge=lfs -text
+OCRFlux-3B.i1-IQ1_S.gguf filter=lfs diff=lfs merge=lfs -text
+OCRFlux-3B.i1-IQ2_M.gguf filter=lfs diff=lfs merge=lfs -text
+OCRFlux-3B.i1-IQ2_S.gguf filter=lfs diff=lfs merge=lfs -text
+OCRFlux-3B.i1-IQ2_XS.gguf filter=lfs diff=lfs merge=lfs -text
+OCRFlux-3B.i1-IQ2_XXS.gguf filter=lfs diff=lfs merge=lfs -text
+OCRFlux-3B.i1-IQ3_M.gguf filter=lfs diff=lfs merge=lfs -text
+OCRFlux-3B.i1-IQ3_S.gguf filter=lfs diff=lfs merge=lfs -text
+OCRFlux-3B.i1-IQ3_XS.gguf filter=lfs diff=lfs merge=lfs -text
+OCRFlux-3B.i1-IQ3_XXS.gguf filter=lfs diff=lfs merge=lfs -text
+OCRFlux-3B.i1-IQ4_NL.gguf filter=lfs diff=lfs merge=lfs -text
+OCRFlux-3B.i1-IQ4_XS.gguf filter=lfs diff=lfs merge=lfs -text
+OCRFlux-3B.i1-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
+OCRFlux-3B.i1-Q2_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+OCRFlux-3B.i1-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
+OCRFlux-3B.i1-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+OCRFlux-3B.i1-Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+OCRFlux-3B.i1-Q4_0.gguf filter=lfs diff=lfs merge=lfs -text
+OCRFlux-3B.i1-Q4_1.gguf filter=lfs diff=lfs merge=lfs -text
+OCRFlux-3B.i1-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+OCRFlux-3B.i1-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+OCRFlux-3B.i1-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+OCRFlux-3B.i1-Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+OCRFlux-3B.i1-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text

OCRFlux-3B.i1-IQ1_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:9953e6b374014ce978b106f131d42856f5fcac7bd076bd70442dfe0181804e52
+size 951590496

OCRFlux-3B.i1-IQ1_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c8182c55eec27e700825c9dfdc47bfd4e292dae3ac974925885b8decc4c87285
+size 892657248

OCRFlux-3B.i1-IQ2_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:2b609b6793471ee332272c901c8bee7136fc01a5197c8355e0e48e4351c4d1b7
+size 1273628256

OCRFlux-3B.i1-IQ2_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:52fc083662fd7fd23f93bfbb2dd0404fc50a2d594fab8299dfd1703cc637d6fa
+size 1195050592

OCRFlux-3B.i1-IQ2_XS.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c11176888d514ca2f74e703d4f803a12614274e8699d84a2dac79e56b9a18dec
+size 1133108832

OCRFlux-3B.i1-IQ2_XXS.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:aad0a5bc496b7815b0a2f75bf8b5e3be5773e0447c0a7a76d65015809589704e
+size 1049812576

OCRFlux-3B.i1-IQ3_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:5b867479a15bbb7c28c263e2dfebb347fdebaca474c1e081d098ec75652e9d0d
+size 1621937760

OCRFlux-3B.i1-IQ3_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c00b6ab48340e9e903428bd7ef538c890ac714b63217d3a5c0e99898a91fed50
+size 1589907040

OCRFlux-3B.i1-IQ3_XS.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:6e809b9a76a9733eddf09a6e0ad092e9a2730bdf8749a8e4f83798f5f77f8b68
+size 1524878944

OCRFlux-3B.i1-IQ3_XXS.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a10ac676498064a5c4a2d101fb07271170f250a683c8d1d1ce9a858b23307813
+size 1415939680

OCRFlux-3B.i1-IQ4_NL.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:8703b3fcb257558981e391f17e4dc49223cacaf02d1dd19315d17ce3deb9181d
+size 1999509088

OCRFlux-3B.i1-IQ4_XS.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:2a5243423cfa4899b3db65e1565b4ef39c2e838850c9a56a2bdf79f299446816
+size 1903687264

OCRFlux-3B.i1-Q2_K.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:421bc5d8a054001ded1c6bc2bbda4ebdab89071d4af9dfad21d85d211fa7715f
+size 1376249440

OCRFlux-3B.i1-Q2_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:45a1bf98fda8aa71e7c9eed76154c3d1d783d85abec3aecf46e31fb4a1a85c9d
+size 1299621472

OCRFlux-3B.i1-Q3_K_L.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:492555c562996b87283a40d4bc4c0efcbcab83aeb80da4b8541e5a48f2696a95
+size 1840434784

OCRFlux-3B.i1-Q3_K_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:5d6e25417e9303c222239905073c06c517691bfcaccce53976a4215b936b58fe
+size 1723518560

OCRFlux-3B.i1-Q3_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f83ba11c8cac22b1d93851632a7d5e1d23bc0d8f79a320e4f0b613f6a9dcf792
+size 1587400288

OCRFlux-3B.i1-Q4_0.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:bd8e305d5d55638f23617dce5d7175a8183daefcb2e913cc7894db010baff977
+size 2002785888

OCRFlux-3B.i1-Q4_1.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:cca9c1f7144857b1ff52954587c8955e9ac97d84469aaec0a55645f921132636
+size 2189973088

OCRFlux-3B.i1-Q4_K_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a32ccc3dd914aa3346b4eb95cc46486210c5c2aeee02a4f6263608ebdbf8abd0
+size 2104202848

OCRFlux-3B.i1-Q4_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:7718b52e7f1dce26dd25defa8f49e590e854e0a5fd350efc93600650e26171d0
+size 2008684128

OCRFlux-3B.i1-Q5_K_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:63d09ffa96b8388a8047ae0e27dcc0df0fac64ad0cbad317762ed0be9c504367
+size 2437944928

OCRFlux-3B.i1-Q5_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:99a54965303e5e8e83cdcb512253ede3396df49294daa38e6476cb9e65693181
+size 2382796384

OCRFlux-3B.i1-Q6_K.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:8ffb7fa83438a7577791059417dc8e90fc9c148b178735ec65c277e0b814118c
+size 2792545888