Enable multimodal (vision): add processor_config.json and update model card

README.md

@@ -3,12 +3,16 @@ language:
 - en
 license: gemma
 library_name: vllm
-pipeline_tag: text-generation
+pipeline_tag: image-text-to-text
+base_model:
+- google/gemma-4-31B-it
 tags:
 - gemma
 - gemma4
 - pearl
 - quantized
+- multimodal
+- vision
 - text-generation
 - mining
 ---
@@ -23,16 +27,18 @@ Pearl **Gemma 4** instruction-tuned checkpoint for **Pearl** inference and minin
 
 ## Benchmarks
 
-Results from our evaluation runs. **Original** is the unmodified Gemma 4 31B instruction model; **Pearl** is this checkpoint (`pearl-ai/Gemma-4-31B-it-pearl`).
+Results from our evaluation runs (lmms-eval + vLLM, full test sets). **Original** is `google/gemma-4-31B-it`; **Pearl** is this checkpoint.
 
-| Model | GPQA | MMLU | HumanEval (pass@1) | MGSM3 |
-| --- | --- | --- | --- | --- |
-| Original | 77.27% | 90.93% | 94.70% | 88.62% |
-| Pearl | 77.37% | 90.56% | 94.15% | 89.09% |
+| Model | GPQA | MMLU | HumanEval (pass@1) | MGSM3 | MMMU-Pro Vision* |
+| --- | --- | --- | --- | --- | --- |
+| Original | 77.27% | 90.93% | 94.70% | 88.62% | 54.57% |
+| Pearl | 77.37% | 90.56% | 94.15% | 89.09% | 54.45% |
+
+\* **MMMU-Pro Vision (Pearl / Original):** lmms-eval `mmmu_pro_vision`, direct-answer prompting, `max_new_tokens=256`, full test set (1730 samples). Google reports **76.9%** on MMMU Pro in the [Gemma 4 model card](https://ai.google.dev/gemma/docs/core/model_card_4); they do not publish the eval recipe for that figure (prompting, subset, or aggregation).
 
 ## Pearl mining (vLLM plugin)
 
-**Pearl mining** means serving through the Pearl **miner stack**: a `pearld` node (RPC), `pearl-gateway`, and the **vLLM miner** build that loads the Pearl plugin (NoisyGEMM / gateway integration). That stack ties matrix work from inference to the chain’s useful-work mining flow. Details and layout are in the [miner README](https://github.com/pearl-research-labs/pearl/tree/master/miner).
+**Pearl mining** means serving through the Pearl **miner stack**: a `pearld` node (RPC), `pearl-gateway`, and the **vLLM miner** build that loads the Pearl plugin (NoisyGEMM / gateway integration). Details are in the [miner README](https://github.com/pearl-research-labs/pearl/tree/master/miner).
 
 Typical flow:
 
@@ -40,15 +46,15 @@ Typical flow:
 2. Start the **Pearl miner / vLLM** image or workspace (plugin-enabled vLLM).
 3. Point the server at this model; gateway + miner components handle **mining-side** integration.
 
-High-level prerequisites (same family as other Pearl model cards):
+High-level prerequisites:
 
-- Python 3.12, **`uv`**, CUDA + NVIDIA GPU (see miner docs for supported architectures, e.g. sm90-class notes there)
+- Python 3.12, **`uv`**, CUDA + NVIDIA GPU (see miner docs for supported architectures)
 - Rust toolchain (for Pearl miner build paths)
 - A running **`pearld`** with RPC credentials for the gateway
 
 ### Docker (recommended for mining)
 
-From the **Pearl** repository root, build and run the miner image (substitute RPC values); use this model id in place of the example model:
+From the **Pearl** repository root:
 
 ```bash
 docker buildx build -t vllm_miner . -f miner/vllm-miner/Dockerfile
@@ -67,12 +73,9 @@ docker run --rm -it --gpus all \
   --host 0.0.0.0 --port 8000 \
   --max-model-len 8192 \
   --gpu-memory-utilization 0.9 \
-  --enforce-eager \
-  --language-model-only
+  --enforce-eager
 ```
 
-For **inference-only** (no mining), or when you have the plugin installed in your own `uv` environment, you can use plain vLLM as below.
-
 ## Inference with vLLM
 
 Serve from the Hugging Face Hub id (or from a local directory containing this snapshot):
@@ -83,21 +86,21 @@ uv run vllm serve pearl-ai/Gemma-4-31B-it-pearl \
   --port 8000 \
   --max-model-len 8192 \
   --gpu-memory-utilization 0.9 \
-  --enforce-eager \
-  --language-model-only
+  --enforce-eager
 ```
 
 **Flags (same as above, one line):**
 
 ```bash
-uv run vllm serve pearl-ai/Gemma-4-31B-it-pearl --host 0.0.0.0 --port 8000 --max-model-len 8192 --gpu-memory-utilization 0.9 --enforce-eager --language-model-only
+uv run vllm serve pearl-ai/Gemma-4-31B-it-pearl --host 0.0.0.0 --port 8000 --max-model-len 8192 --gpu-memory-utilization 0.9 --enforce-eager
 ```
 
-From a **local** checkout of this repo (e.g. after `git clone` or copying files), point `vllm serve` at the directory path instead of the Hub id.
+These commands load the full model (text + vision). Append `--language-model-only` if you only need text — it disables the vision tower and uses less GPU memory.
 
 ## Model details
 
 - **Architecture:** `Gemma4ForConditionalGeneration` (`model_type`: `gemma4`)
+- **Modalities:** Text, Image
 
 ## License
 

processor_config.json

@@ -0,0 +1,75 @@
+{
+  "audio_ms_per_token": 40,
+  "audio_seq_length": 750,
+  "feature_extractor": {
+    "dither": 0.0,
+    "feature_extractor_type": "Gemma4AudioFeatureExtractor",
+    "feature_size": 128,
+    "fft_length": 512,
+    "fft_overdrive": false,
+    "frame_length": 320,
+    "hop_length": 160,
+    "input_scale_factor": 1.0,
+    "max_frequency": 8000.0,
+    "mel_floor": 0.001,
+    "min_frequency": 0.0,
+    "padding_side": "right",
+    "padding_value": 0.0,
+    "per_bin_mean": null,
+    "per_bin_stddev": null,
+    "preemphasis": 0.0,
+    "preemphasis_htk_flavor": true,
+    "return_attention_mask": true,
+    "sampling_rate": 16000
+  },
+  "image_processor": {
+    "do_convert_rgb": true,
+    "do_normalize": false,
+    "do_rescale": true,
+    "do_resize": true,
+    "image_mean": [
+      0.0,
+      0.0,
+      0.0
+    ],
+    "image_processor_type": "Gemma4ImageProcessor",
+    "image_seq_length": 280,
+    "image_std": [
+      1.0,
+      1.0,
+      1.0
+    ],
+    "max_soft_tokens": 280,
+    "patch_size": 16,
+    "pooling_kernel_size": 3,
+    "resample": 3,
+    "rescale_factor": 0.00392156862745098
+  },
+  "image_seq_length": 280,
+  "processor_class": "Gemma4Processor",
+  "video_processor": {
+    "do_convert_rgb": true,
+    "do_normalize": true,
+    "do_rescale": true,
+    "do_resize": true,
+    "do_sample_frames": true,
+    "image_mean": [
+      0.0,
+      0.0,
+      0.0
+    ],
+    "image_std": [
+      1.0,
+      1.0,
+      1.0
+    ],
+    "max_soft_tokens": 70,
+    "num_frames": 32,
+    "patch_size": 16,
+    "pooling_kernel_size": 3,
+    "resample": 3,
+    "rescale_factor": 0.00392156862745098,
+    "return_metadata": false,
+    "video_processor_type": "Gemma4VideoProcessor"
+  }
+}