Update README.md

README.md

@@ -1,3 +1,221 @@
----
-license: apache-2.0
----
+---
+license: apache-2.0
+base_model:
+- tifa-benchmark/llama2_tifa_question_generation
+pipeline_tag: text-generation
+tags:
+  - text-to-image-evaluation
+  - faithfulness
+  - lora
+  - tifa
+  - llama2
+  - question-generation
+language: en
+---
+
+# Llama2-TIFA-AWQ
+
+## Model Description
+
+Llama2-TIFA-AWQ is a fine-tuned and AWQ-quantized version of [tifa-benchmark/llama2_tifa_question_generation](https://huggingface.co/tifa-benchmark/llama2_tifa_question_generation) that addresses structural limitations in the original TIFA question generation model while providing **significantly faster inference**. The model combines structural refinement with AWQ quantization to achieve optimal performance-speed balance for TIFA question generation.
+
+## Key Innovation: Fixing Structural Issues
+
+This model represents a **refinement approach** rather than training from scratch:
+- **Original model strength**: Deep TIFA domain knowledge and question generation capabilities
+- **Original model issues**: 
+  - Generated multiple questions for the same attribute
+  - Lacked negative verification questions
+- **This solution**: Structural fine-tuning to enforce 4-question format while preserving domain expertise
+
+## AWQ Quantization: Performance Optimization
+
+This model includes **AWQ (Activation-aware Weight Quantization)** for optimal inference speed:
+
+### Performance Comparison
+- **SmolLM2 models**: ~3 seconds (baseline small models)
+- **Original LLaMA 2**: ~20 seconds (full precision)
+- **Llama2-TIFA-AWQ**: ~8 seconds (**2.5x faster** than full precision)
+
+## Intended Use
+
+This model generates exactly 4 structured visual verification questions for text-to-image evaluation:
+- **Mixed question types**: Both yes/no and multiple choice questions
+- **Comprehensive coverage**: Colors, shapes, objects, materials, spatial relationships
+- **Balanced verification**: Both positive presence and negative absence testing
+- **Controlled structure**: Exactly 4 questions without redundancy
+
+## Model Details
+
+- **Base Model**: tifa-benchmark/llama2_tifa_question_generation (LLaMA 2 architecture)
+- **Model Size**: ~7B parameters (AWQ quantized)
+- **Fine-tuning Method**: LoRA (Low-Rank Adaptation) for structural refinement
+- **Quantization**: AWQ (Activation-aware Weight Quantization)
+- **Training Framework**: Transformers + TRL + PEFT
+- **License**: apache-2.0
+
+## Training Details
+
+### Structural Refinement Configuration
+- **Training Method**: Supervised Fine-Tuning with LoRA on pre-trained TIFA model
+- **LoRA Configuration**:
+  - r: 32
+  - lora_alpha: 64
+  - lora_dropout: 0.05
+  - Target modules: `["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]`
+  
+- **Training Parameters**:
+  - Epochs: 2 
+  - Learning Rate: 3e-4
+  - Batch Size: 6 
+  - Gradient Accumulation: 5 steps
+  - Max Sequence Length: 768
+  - LR Scheduler: Cosine with 5% warmup
+  - Precision: FP16
+
+### Enhanced Dataset
+- **Size**: 18,000 examples with structured 4-question format
+- **Focus**: Teaching proper question structure and negative verification
+- **Validation**: Category-balanced split ensuring robust evaluation
+- **Format**: LLaMA 2 chat template with preprocessed text format
+
+## Usage
+
+### Installation
+
+```bash
+pip install transformers torch autoawq
+```
+
+### Basic Usage
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
+import torch
+
+model_path = "kawchar85/Llama2-TIFA-AWQ"
+
+# Load AWQ quantized model and tokenizer
+tokenizer = AutoTokenizer.from_pretrained(model_path)
+model = AutoModelForCausalLM.from_pretrained(
+    model_path, 
+    torch_dtype=torch.float16,
+    device_map="auto"
+)
+
+# Create pipeline (optimized for ~8 second inference)
+pipe = pipeline(
+    "text-generation",
+    model=model,
+    tokenizer=tokenizer,
+    max_new_tokens=512,
+    do_sample=True,
+    temperature=0.7,
+    top_p=0.9,
+)
+
+# System prompt for TIFA question generation
+system_msg = """\
+You are a TIFA (Text-to-Image Faithfulness evaluation with question Answering) question generator. Given an image description, create exactly 4 visual verification questions with multiple choice answers. Each question should test different visual aspects that can be verified by looking at the image.
+
+Guidelines:
+- Focus on colors, shapes, objects, materials, spatial relationships, and other visually verifiable elements
+- Mix yes/no questions (2 choices: "no", "yes") and multiple choice questions (4 choices)
+- Each question should test a DIFFERENT aspect of the description
+- Ensure questions can be answered by visual inspection of the image
+- Use elements explicitly mentioned in the description
+- Include both positive verification (testing presence, answer: "yes") and negative verification (testing absence, answer: "no")
+- Make distractors realistic and relevant to the domain
+
+Format each question as:
+Q[number]: [question text]
+C: [comma-separated choices]
+A: [correct answer]
+
+Generate questions that test visual faithfulness between the description and image."""
+
+# Generate evaluation questions
+description = "a lighthouse overlooking the ocean"
+prompt = (
+    "<s>[INST] <<SYS>>\n"
+    f"{system_msg}\n"
+    "<</SYS>>\n\n"
+    f'Create 4 visual verification questions for this description: "{description}" [/INST]'
+)
+
+output = pipe(prompt)[0]['generated_text']
+response = output[len(prompt):]
+print(response)
+```
+
+### Example Output
+
+**For "a lighthouse overlooking the ocean":**
+```
+Q1: What type of structure is prominently featured in the image?
+C: windmill, lighthouse, castle, tower
+A: lighthouse
+
+Q2: What body of water is the lighthouse overlooking?
+C: lake, river, ocean, pond
+A: ocean
+
+Q3: Are there any mountains visible in the scene?
+C: no, yes
+A: no
+
+Q4: Is the lighthouse positioned to overlook a body of water?
+C: no, yes
+A: yes
+```
+
+## Advantages Over Original Model
+
+### Structural Improvements
+- **Controlled output**: Exactly 4 questions instead of variable numbers
+- **No redundancy**: Eliminates multiple questions per attribute
+- **Negative verification**: Includes proper negative questions (absent elements)
+
+### Preserved Strengths
+- **Domain expertise**: Retains deep TIFA knowledge from original training
+- **Question quality**: Maintains high-quality question formulation
+- **Visual focus**: Strong emphasis on verifiable visual elements
+- **Natural language**: Experienced question generation capabilities
+
+## Comparison with SmolLM2 Series
+
+| Aspect | Llama2-TIFA-AWQ | SmolLM2 Series |
+|--------|-----------------|----------------|
+| **Starting point** | TIFA-specialized model | General instruction models |
+| **Domain knowledge** | ✅ Pre-existing TIFA expertise | ⭐ Learned during fine-tuning |
+| **Model size** | ~7B parameters (AWQ quantized) | 135M - 1.7B parameters |
+| **Inference speed** | ~8 seconds (AWQ optimized) | ~3 seconds (small models) |
+| **Training approach** | Structural refinement + quantization | Full task learning |
+| **Memory efficiency** | ✅ AWQ quantized | ⭐ Naturally smaller |
+| **Question quality** | ✅ Deep domain knowledge | ⭐ Systematic structure |
+
+
+## Citation
+
+```bibtex
+@misc{llama2-tifa-refined-2025,
+  title={Llama2-TIFA: Structural Refinement of LLaMA 2 for Text-to-Image Faithfulness Assessment},
+  author={kawchar85},
+  year={2025},
+  url={https://huggingface.co/kawchar85/Llama2-TIFA-AWQ},
+  note={Fine-tuned from tifa-benchmark/llama2_tifa_question_generation}
+}
+```
+
+## Model Ecosystem
+
+This model complements the broader TIFA question generation ecosystem:
+
+**Specialized TIFA models:**
+- **Llama2-TIFA** ← *You are here* (Domain expert, refined structure)
+
+**General→TIFA models:**
+- **[SmolLM2-135M-Instruct-TIFA](https://huggingface.co/kawchar85/SmolLM2-135M-Instruct-TIFA)**: Compact version
+- **[SmolLM2-360M-Instruct-TIFA](https://huggingface.co/kawchar85/SmolLM2-360M-Instruct-TIFA)**: Balanced version  
+- **[SmolLM2-1.7B-Instruct-TIFA](https://huggingface.co/kawchar85/SmolLM2-1.7B-Instruct-TIFA)**: Structured version
+- **[SmolLM2-1.7B-Instruct-TIFA-Random](https://huggingface.co/kawchar85/SmolLM2-1.7B-Instruct-TIFA-Random)**: Flexible version