Update README.md

README.md

@@ -40,21 +40,32 @@ Zamba2-VL-2.7B performs strongly against models of comparable size and inference
 
 All numbers are run on the Zyphra evaluation harness (based on VLMEvalKit). Other models are ordered by total parameter count. Bold indicates the best score in each row, while <u>underlined</u> values indicate the lowest score.
 
+
 ## Quick start
 
 ### Prerequisites
 
-To use Zamba2-VL, install `transformers` from source:
+To use Zamba2-VL, install `zamba2-vl` branch from our fork of `transformers` library, which is based on the v4.57.1 of `transformers`:  
+```bash
+pip install "transformers @ git+https://github.com/Zyphra/transformers.git@zamba2-vl"
+pip install qwen-vl-utils==0.0.2
+pip install flash_attn
+```
+
+The command above relies on requirements for `transformers v4.57.1` being installed in your environment. If you're installing in a fresh Python environment, you might want to specify a specific extra, like `[dev-torch]`, to install all the dependencies:  
+```bash
+pip install "transformers[dev-torch] @ git+https://github.com/Zyphra/transformers.git@zamba2-vl"
+```
+
+For the fastest setup, ensure your environment matches an existing flash_attn wheel, otherwise the installation will build from source.
 
-1. `git clone https://github.com/huggingface/transformers.git`
-2. `cd transformers && pip install .`
  
 To install dependencies necessary to run Mamba2 kernels, install `mamba-ssm` from source (due to compatibility issues with PyTorch) as well as `causal-conv1d`:
 
-1. `git clone https://github.com/state-spaces/mamba.git`
-2. `cd mamba && git checkout v2.1.0 && pip install .`
-3. `pip install causal-conv1d`
-
+```bash
+pip install --no-build-isolation "causal-conv1d @ git+https://github.com/Zyphra/z-causal-conv1d.git@zamba2-vl"
+pip install --no-build-isolation "mamba-ssm @ git+https://github.com/Zyphra/mamba.git@zamba2-vl"
+```
 
 You can run the model without using the optimized Mamba2 kernels, but it is **not** recommended as it will result in significantly higher latency and memory usage. 
 
@@ -62,15 +73,33 @@ You can run the model without using the optimized Mamba2 kernels, but it is **no
 ### Inference
 
 ```python
-from transformers import AutoTokenizer, AutoModelForCausalLM
+from transformers import Zamba2_VLForConditionalGeneration, Zamba2_VLProcessor
 import torch
-
-tokenizer = AutoTokenizer.from_pretrained("Zyphra/Zamba2-VL-2.7B")
-model = AutoModelForCausalLM.from_pretrained("Zyphra/Zamba2-VL-2.7B", device_map="cuda", torch_dtype=torch.bfloat16)
-
-input_text = "What factors contributed to the fall of the Roman Empire?"
-input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")
-
-outputs = model.generate(**input_ids, max_new_tokens=100)
-print(tokenizer.decode(outputs[0]))
+from PIL import Image
+from qwen_vl_utils import process_vision_info
+import requests
+
+device = "cuda"
+processor = Zamba2_VLProcessor.from_pretrained("Zyphra/Zamba2-VL-2.7B", temporal_patch_size=1)
+model = Zamba2_VLForConditionalGeneration.from_pretrained("Zyphra/Zamba2-VL-2.7B", device_map=device, torch_dtype=torch.bfloat16, attn_implementation="flash_attention_2")
+
+url = "http://images.cocodataset.org/val2017/000000039769.jpg"
+image = Image.open(requests.get(url, stream=True).raw)
+question = "What do you see in the image? Give us some detail."
+num_img_tokens = 3400
+
+conversation = [
+    {"role": "user", "content": [
+        {"type": "image", "image": image, "max_pixels" : num_img_tokens * 28 * 28, "min_pixels" : 10 * 28 * 28},
+        {"type": "text", "text": question},
+      ]
+    },
+]
+prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
+images, _ = process_vision_info(conversation)
+inputs = processor(text=prompt, images=images, add_special_tokens=True, return_tensors="pt")
+inputs = {key: value.to(device) for key, value in inputs.items()}
+
+outputs = model.generate(**inputs, max_new_tokens=100)
+print(processor.tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))
 ```