litert-community

/

gemma-4-E2B-it-litert-lm

like 412

Follow

LiteRT Community (FKA TFLite) 8.74k

Thanks for the feedback, and will look at the requirements.

If anybody else in the community have time and is willing to help, please jump in o help. It should be easy to convert using LiteRT torch, and just need to run some eval to make sure the converted model works resonably well.

​I have the hardware for this and can take a look. I'll see if I can get it converted and tested for you.

it should easy convert to liteRT

Hi litert-community,

I'm building an offline-first Android health platform for Sierra Leone, and I'm trying to run MedGemma on-device using LiteRT-LM.

I noticed the community has done a great job converting Gemma variants to .litertlm format, but I couldn't find a converted version of google/medgemma-1.5-4b-it. Would anyone be willing to add it to the litert-community repo?

The use case is health Q&A for users in a low-connectivity environment — MedGemma's medical training makes it a much better fit than a general Gemma model for this. I don't have the local hardware to run the conversion myself, so a community-hosted version would be a huge help.

Happy to test it and report back on Android performance if that's useful to anyone.

Thanks!

i've converted the model, will be uploading now

the model should run on devices with around 4gb of ram

you can contact me for more info or request regarding models :)
@larrybah
its live at https://huggingface.co/litert-community/MedGemma-1.5-4B-IT

Thank you very much @TobiasLogic and everyone in the community.

@larrybah you're welcome, if you need any help that requires compute just send a text here 😊

@TobiasLogic would you be able to create a model that supports the gpu backend as well?

yep sure, il make it in a bit

Done added a GPU-backend variant for MedGemma-1.5-4B-IT: medgemma-1.5-4b-it_q8_gpu_ekv2048.litertlm. Used the standard int8 dynamic recipe (dynamic_wi8_afp32) with GPU dynamic prefill/cache enabled, which is the recipe Google documents as CPU+GPU compatible (the original file used a more aggressive int4 block32 scheme picked for the 4GB-RAM constraint, which isn't GPU-delegate friendly). Confirmed it loads and generates correctly

I'm getting only pad tags as the output when running with gpu backend. This is on an RTX A500 (4GB)

> litert-lm run --backend=gpu medgemma-1.5-4b-it_q8_gpu_vision_ekv2048.litertlm --prompt="How do you differentiate bacterial from viral pneumonia?"
<pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad>

No regression with the CPU backend compared to other models.

> litert-lm run --backend=cpu medgemma-1.5-4b-it_q8_gpu_vision_ekv2048.litertlm --prompt="How do you differentiate bacterial from viral pneumonia?"`
Differentiating bacterial pneumonia from viral pneumonia can be challenging as symptoms can overlap significantly. However, there are several key differences 
and diagnostic approaches that help distinguish between the two:
...

Found the actual cause of the garbage output and pushed a fix. MedGemma's output projection matrix (262208 vocab x 2560 hidden, quantized int8) comes out to 640MB as a single tensor. WebGPU's storage buffer binding limit is 128MB. The delegate fails that validation silently instead of erroring, and that's what produced the all pad token output. I confirmed this by building the native litert_lm_main runtime from source and reproducing the exact validation error.

Fix: split the lm_head weight into 8 chunks before export so each tensor stays under the limit. Verified the validation error is gone and the model still generates correctly on CPU. New file is up: medgemma-1.5-4b-it_q8_gpu_ekv2048.litertlm (this replaces the earlier broken upload).

I couldn't get a full end to end GPU run on my side since I only have software rendered Vulkan available in this environment, which is far too slow to be a real test. @mistermadge would you be able to try this one on your RTX A500? Given the root cause is now identified and fixed rather than guessed at, I'm fairly confident this one will actually work, but a real confirmation from your hardware would settle it.

Okey, i will check the requirements and how to do it !