Hugging Face's logo

sphaela
/
gemma-4-E2B-it-AutoRound-GGUF

GGUF
multilingual
auto-round
intel
quantization
vlm
conversational
Model card Files
xet
 Community
sphaela commited on
Commit
6544294
·
verified ·
1 Parent(s): 73ba94e

Add files using upload-large-folder tool

Browse files
Files changed (20) hide show
  1. .gitattributes +18 -0
  2. README.md +67 -3
  3. gemma-4-E2B-it-Q2_K_MIXED.gguf +3 -0
  4. gemma-4-E2B-it-Q2_K_S.gguf +3 -0
  5. gemma-4-E2B-it-Q3_K_L.gguf +3 -0
  6. gemma-4-E2B-it-Q3_K_M.gguf +3 -0
  7. gemma-4-E2B-it-Q3_K_S.gguf +3 -0
  8. gemma-4-E2B-it-Q4_0.gguf +3 -0
  9. gemma-4-E2B-it-Q4_1.gguf +3 -0
  10. gemma-4-E2B-it-Q4_K_M.gguf +3 -0
  11. gemma-4-E2B-it-Q4_K_S.gguf +3 -0
  12. gemma-4-E2B-it-Q5_0.gguf +3 -0
  13. gemma-4-E2B-it-Q5_1.gguf +3 -0
  14. gemma-4-E2B-it-Q5_K_M.gguf +3 -0
  15. gemma-4-E2B-it-Q5_K_S.gguf +3 -0
  16. gemma-4-E2B-it-Q6_K.gguf +3 -0
  17. gemma-4-E2B-it-Q8_0.gguf +3 -0
  18. mmproj-model-bf16.gguf +3 -0
  19. mmproj-model-f16.gguf +3 -0
  20. mmproj-model-f32.gguf +3 -0
.gitattributes CHANGED
@@ -33,3 +33,21 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ mmproj-model-f32.gguf filter=lfs diff=lfs merge=lfs -text
37
+ mmproj-model-bf16.gguf filter=lfs diff=lfs merge=lfs -text
38
+ mmproj-model-f16.gguf filter=lfs diff=lfs merge=lfs -text
39
+ gemma-4-E2B-it-Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
40
+ gemma-4-E2B-it-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
41
+ gemma-4-E2B-it-Q5_1.gguf filter=lfs diff=lfs merge=lfs -text
42
+ gemma-4-E2B-it-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
43
+ gemma-4-E2B-it-Q2_K_MIXED.gguf filter=lfs diff=lfs merge=lfs -text
44
+ gemma-4-E2B-it-Q2_K_S.gguf filter=lfs diff=lfs merge=lfs -text
45
+ gemma-4-E2B-it-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
46
+ gemma-4-E2B-it-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
47
+ gemma-4-E2B-it-Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
48
+ gemma-4-E2B-it-Q4_0.gguf filter=lfs diff=lfs merge=lfs -text
49
+ gemma-4-E2B-it-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
50
+ gemma-4-E2B-it-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
51
+ gemma-4-E2B-it-Q4_1.gguf filter=lfs diff=lfs merge=lfs -text
52
+ gemma-4-E2B-it-Q5_0.gguf filter=lfs diff=lfs merge=lfs -text
53
+ gemma-4-E2B-it-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -1,3 +1,67 @@
1
- ---
2
- license: apache-2.0
3
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ language:
4
+ - en
5
+ base_model: google/gemma-4-E2B-it
6
+ tags:
7
+ - auto-round
8
+ - intel
9
+ - gguf
10
+ - quantization
11
+ - vlm
12
+ ---
13
+
14
+ # Gemma-4-E2B-it GGUF (AutoRound Quantized)
15
+
16
+ This repository contains GGUF quantized versions of [google/gemma-4-E2B-it](https://huggingface.co/google/gemma-4-E2B-it) created using Intel's [AutoRound](https://github.com/intel/auto-round) quantization method.
17
+
18
+ ## Quantization Details
19
+
20
+ The models were quantized using various schemes provided by the `auto-round` tool. For better compatibility and smaller size, we provide unified multimodal projector (`mmproj`) files in F16, BF16, and F32 formats.
21
+
22
+ ### Files and Sizes
23
+
24
+ | File Name | Quant Type | Size | Description |
25
+ |-----------|------------|------|-------------|
26
+ | `gemma-4-E2B-it-Q2_K_S.gguf` | Q2_K_S | 2.7 GB | Extremely high compression, significant quality loss. |
27
+ | `gemma-4-E2B-it-Q2_K_MIXED.gguf` | Q2_K_MIXED | 3.8 GB | **Recommended** high-compression option. Uses Q4 for KV cache with good quality. |
28
+ | `gemma-4-E2B-it-Q3_K_S.gguf` | Q3_K_S | 2.9 GB | Very high compression, notable quality loss. |
29
+ | `gemma-4-E2B-it-Q3_K_M.gguf` | Q3_K_M | 2.9 GB | Balanced 3-bit quantization. |
30
+ | `gemma-4-E2B-it-Q3_K_L.gguf` | Q3_K_L | 2.9 GB | High quality 3-bit quantization. |
31
+ | `gemma-4-E2B-it-Q4_0.gguf` | Q4_0 | 3.2 GB | Standard 4-bit quantization, good balance. |
32
+ | `gemma-4-E2B-it-Q4_1.gguf` | Q4_1 | 3.3 GB | Higher quality 4-bit quantization than Q4_0. |
33
+ | `gemma-4-E2B-it-Q4_K_S.gguf` | Q4_K_S | 3.2 GB | Small 4-bit K-quant, good efficiency. |
34
+ | `gemma-4-E2B-it-Q4_K_M.gguf` | Q4_K_M | 3.2 GB | **Recommended** 4-bit K-quant, excellent balance. |
35
+ | `gemma-4-E2B-it-Q5_0.gguf` | Q5_0 | 3.4 GB | Standard 5-bit quantization, very high quality. |
36
+ | `gemma-4-E2B-it-Q5_1.gguf` | Q5_1 | 3.5 GB | Higher quality 5-bit quantization than Q5_0. |
37
+ | `gemma-4-E2B-it-Q5_K_S.gguf` | Q5_K_S | 3.4 GB | Small 5-bit K-quant, very high quality. |
38
+ | `gemma-4-E2B-it-Q5_K_M.gguf` | Q5_K_M | 3.4 GB | **Recommended** 5-bit K-quant, near-lossless. |
39
+ | `gemma-4-E2B-it-Q6_K.gguf` | Q6_K | 3.6 GB | 6-bit K-quant, virtually indistinguishable from F16. |
40
+ | `gemma-4-E2B-it-Q8_0.gguf` | Q8_0 | 4.7 GB | 8-bit quantization, near-lossless. |
41
+ | `mmproj-model-f16.gguf` | F16 | 940 MB | Unified Projector in Float16 format. |
42
+ | `mmproj-model-bf16.gguf` | BF16 | 942 MB | Unified Projector in BFloat16 format. |
43
+ | `mmproj-model-f32.gguf` | F32 | 1.8 GB | Unified Projector in Float32 format. |
44
+
45
+ ## Generate the Model
46
+
47
+ The models were generated using Intel's AutoRound with the following command:
48
+
49
+ ```bash
50
+ auto-round --model google/gemma-4-E2B-it --output_dir ./quantized/ --scheme <SCHEME> --iters 0
51
+ ```
52
+
53
+ **Note:** To reproduce this quantization, you need the following Pull Request:
54
+ - [Intel AutoRound PR #1655](https://github.com/intel/auto-round/pull/1655)
55
+ - And the specific fix mentioned in [this comment](https://github.com/intel/auto-round/pull/1655#issuecomment-4259531072).
56
+
57
+ ## Usage with llama.cpp
58
+
59
+ These models can be used with `llama.cpp`. For multimodal usage, you must specify the projector file:
60
+
61
+ ```bash
62
+ ./llama-cli -m gemma-4-E2B-it-Q4_K_M.gguf --mmproj mmproj-model-f16.gguf --image your_image.jpg -p "Describe this image."
63
+ ```
64
+
65
+ ## About AutoRound
66
+
67
+ [AutoRound](https://github.com/intel/auto-round) is an advanced quantization technique from Intel that aims to minimize accuracy loss through automated rounding optimization.
gemma-4-E2B-it-Q2_K_MIXED.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ecebe017210864800e36c8aa5abca900fd850ba878ff208d1de94b8263f97426
3
+ size 4002320928
gemma-4-E2B-it-Q2_K_S.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:86ed05c2b2d22f5f088f94157650dd90edb2ed4717887217fbf8242adfdc8000
3
+ size 2896071200
gemma-4-E2B-it-Q3_K_L.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:78989b77e7e7c3e83062d62dab76de64bb7480fb21dbb71a064474ca59cd27d5
3
+ size 3088602656
gemma-4-E2B-it-Q3_K_M.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:72ee9d8d33934c2e13f8fa885f874b8c5128dc90add842df44a19970149d0b71
3
+ size 3088602656
gemma-4-E2B-it-Q3_K_S.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:520e9a6dfd465e053f6955271e60d9073b5eb291e550d3ed9ccf66f5bd68a69d
3
+ size 3088602656
gemma-4-E2B-it-Q4_0.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:735ad48619403535628a01d35402a1b41b92d2b9cea1d78e388b9bb3be75f1ff
3
+ size 3340374560
gemma-4-E2B-it-Q4_1.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ca59c2030e37b34d2c904271185982260cbf86cd53ee8ef414b06779900504fd
3
+ size 3458855456
gemma-4-E2B-it-Q4_K_M.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b0dd54a3d04f4b328334b629b84b2c624cab34f643b529b4032b7278ea379f12
3
+ size 3340374560
gemma-4-E2B-it-Q4_K_S.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:dd95f6cc08f323ae7ae606404ad5c39adfd5946b4a59ab1e150d68c4ba85bfaa
3
+ size 3340374560
gemma-4-E2B-it-Q5_0.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2da61e5daecd3fdde101347b59b61a6f2432090ba54c39ff842861dfcb524530
3
+ size 3577336352
gemma-4-E2B-it-Q5_1.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1e1699b47f145c2c1e781b1152884c14c57e51b9fe1b662e9b13eba9e309e9be
3
+ size 3695817248
gemma-4-E2B-it-Q5_K_M.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:cf7ca17f4d5f94992a053dba15827b926cd4cbd48a12b3df2424c61faec4413a
3
+ size 3577336352
gemma-4-E2B-it-Q5_K_S.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7db5a527ff2594a9ca21338088ccc245a53c56d4246217ef671e4b9695ecdd8d
3
+ size 3577336352
gemma-4-E2B-it-Q6_K.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1fd868281bee432f9f2374c389fae963320e9e079ee22901d8e5b5fc2d4a274d
3
+ size 3829108256
gemma-4-E2B-it-Q8_0.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e4e4f4c5ea6ab1cad3044628d2869bd4f55ac506ce8b3a6fd0ca560c51caa4b0
3
+ size 4954591776
mmproj-model-bf16.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b6f027287e80ff4ced6ae4a0ccbe9623ffc66399250bdd5502cd209b3a3d4f52
3
+ size 986833248
mmproj-model-f16.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ecfffad417cd69f8e9071319d0849bf8caa4d7c6ea4e4115432596563d53d397
3
+ size 985653600
mmproj-model-f32.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e53b222bdbf1b00ecb18beda044579e7640b09e9160da96dada02e8bc45e4a79
3
+ size 1903026528