---
base_model: Qwen/Qwen3-VL-4B-Instruct
library_name: peft
pipeline_tag: image-text-to-text
tags:
  - qwen3-vl
  - lora
  - gguf
  - food-recognition
  - nutrition
---

# Beforeat Food Nutrition Vision LoRA

This repository contains one LoRA adapter for `Qwen/Qwen3-VL-4B-Instruct` in
PEFT and llama.cpp GGUF formats. Version 2 prioritizes Food-101 dish recognition
and adds experimental visible-ingredient and rough-gram output.

Nutrition totals are **not** predicted directly by this adapter. Beforeat uses
the recognized ingredients and estimated portions as inputs to a separate USDA
FoodData Central lookup and rules layer for calories and macronutrients.

## Current Files

- `adapter_model.safetensors`: v2 PEFT weights
- `adapter_config.json`: rank 8, alpha 16 PEFT configuration
- `beforeat-food-nutrition-vision-lora-v2-f16.gguf`: v2 F16 llama.cpp adapter
- `blend_metadata.json`: reproducible v2 weight-interpolation metadata
- `beforeat-food-nutrition-vision-lora-f16.gguf`: historical v1 GGUF release

Only one adapter is loaded at runtime. The v1 file remains available for
rollback; new integrations should use the versioned v2 file.

## Prompts

Dish recognition:

```text
Identify the dish in this image. Reply with only the exact Food-101 class name.
```

Experimental visible ingredients and portions:

```text
Estimate the visible ingredients and their portions in grams. Reply only with a JSON array of objects using keys "name" and "grams".
```

## Evaluation

| Task | Evaluation | Result |
|---|---|---:|
| Food-101 dish recognition | 500-image held-out set, strict | 438/500 (87.60%) |
| Food-101 dish recognition | Alias/near matching | 440/500 (88.00%) |
| Visible ingredients | Nutrition5k, 100 images | 44.51% mean recall |
| Matched ingredient portions | Nutrition5k, 100 images | 52.93% mean recall |
| Matched ingredient grams | Nutrition5k, 85 parsed rows | 43.34 g MAE |

The previous food-only release scored 434/500 (86.80%) strict and 438/500
(87.60%) with aliases on the same Food-101 evaluation.

The ingredient/portion results are experimental and should not be interpreted
as direct nutrition accuracy. Hidden ingredients, cooking oils, recipe details,
and true serving mass cannot be reliably recovered from one image.

## Adapter Construction

The v2 weights are a linear interpolation between the 2,500-step Food-101 LoRA
and the continued 4,000-step ingredient/portion LoRA. The nutrition checkpoint
weight is `0.25`. Both inputs use the same rank, alpha, tensor names, and target
modules, producing a single 504-tensor LoRA adapter.

- Rank: `8`
- Alpha: `16`
- Target modules: `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`,
  `up_proj`, `down_proj`

## llama.cpp GGUF Adapter

- Base model: `Qwen/Qwen3-VL-4B-Instruct`
- Required base GGUF: `Qwen3-VL-4B-Instruct-UD-Q4_K_XL.gguf`
- Base GGUF source: `unsloth/Qwen3-VL-4B-Instruct-GGUF`
- Required projector: `mmproj-F16.gguf` from the same base GGUF repository
- Recommended adapter scale: `1.0`
- llama.cpp commit: `00fa7cb284cbf133fc426733bd64238a3588a33e`
- GGUF tensor count: `504`
- GGUF file size: `33065152` bytes
- SHA-256: `ac31fb420d91fd8a4149e694c4da540db6208889c53f86b383069a6422a01f62`

Conversion command:

```bash
PYTHONPATH=tools/llama.cpp \
python tools/llama.cpp/convert_lora_to_gguf.py \
  adapters/qwen3vl4b_food_nutrition_blend_w025 \
  --base models/Qwen3-VL-4B-Instruct-bf16-remapped \
  --outfile adapters/qwen3vl4b_food_nutrition_blend_w025/beforeat-food-nutrition-vision-lora-v2-f16.gguf \
  --outtype f16 \
  --verbose
```

Runtime command:

```bash
llama-mtmd-cli \
  -m Qwen3-VL-4B-Instruct-UD-Q4_K_XL.gguf \
  --mmproj mmproj-F16.gguf \
  --image food.jpg \
  --lora-scaled beforeat-food-nutrition-vision-lora-v2-f16.gguf:1.0 \
  -p "Identify the dish in this image. Reply with only the exact Food-101 class name." \
  --jinja --temp 0 -n 16 -c 4096 --image-min-tokens 1024
```

Validation used the same command with `--device none -ngl 0
--no-mmproj-offload --no-op-offload --no-kv-offload --fit off` after the local
Metal command queue was unavailable. llama.cpp accepted all adapter tensors with
no missing, incompatible, or unexpected tensor errors. The Unsloth
`UD-Q4_K_XL` base and its existing `mmproj-F16.gguf` are compatible and reusable.

## Runtime Spot Checks

| Expected | PyTorch/PEFT v2 | Base GGUF | GGUF + v2 LoRA |
|---|---|---|---|
| prime rib | prime rib | roast beef | prime rib |
| bread pudding | panna cotta | dessert | panna cotta |
| chocolate cake | chocolate mousse | Chocolate Cake | chocolate mousse |

The matching PEFT and GGUF+LoRA predictions demonstrate conversion fidelity,
including the same errors.

## Limitations and Use

- Food recognition is specialized around the 101 Food-101 classes and may not
  generalize to arbitrary dishes, brands, mixed plates, or regional variants.
- Ingredient names and grams are rough visual estimates, not measured portions.
- Use a separate food-composition database and calculation layer for nutrition.
- Food-101 and Nutrition5k were used for research and prototyping; review their
  terms before commercial training or redistribution workflows.
- This adapter is not medical or dietary advice.