Update query_clarification/README.md

query_clarification/README.md

@@ -11,7 +11,7 @@ library_name: transformers
 
 ## Model Summary
 
-**Query Clarification** is an adpater designed for conversational use cases where user queries may be ill-formed, unclear, or have multiple valid interpretations based on the underlying system or content. Use cases include RAG systems, routing, and tool invocation scenarios. We provide experimental results showing that the adapter achieves significantly higher classification accuracy than prompted out-of-the-box models, including frontier models such as gpt-4o. The adapter released here works with the IBM granite-4.0-micro model and is specifically fine-tuned for the following task
+**Query Clarification** is an adapter designed for conversational use cases where user queries may be ill-formed, unclear, or have multiple valid interpretations based on the underlying system or content. Use cases include RAG systems, routing, and tool invocation scenarios. We provide experimental results showing that the adapter achieves significantly higher classification accuracy than prompted out-of-the-box models, including frontier models such as gpt-4o. The adapter released here works with the IBM granite-4.0-micro model and is specifically fine-tuned for the following task:
 
     Given a multi-turn conversation between a user and an AI assistant (and optionally relevant content such as
     RAG documents), detect whether the last user query is underspecified (no clear interpretation or multiple
@@ -20,7 +20,7 @@ library_name: transformers
 - **Developer:** IBM Research
 - **HF Collection:** [Granite Libraries](https://huggingface.co/collections/ibm-granite/granite-libraries)
 - **GitHub Repository:** https://github.com/ibm-granite
-- **Release Date:** March 18th, 2026
+- **Release Date:** April 2026
 - **Model type:** LoRA adapter for [ibm-granite/granite-4.0-micro](https://huggingface.co/ibm-granite/granite-4.0-micro)
 - **License:** [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0)
 
@@ -68,7 +68,7 @@ In this example, the adapter detects that the user's question is underspecified
 
 ## Usage Examples
 
-The recommended way to call this adapter is through the [Mellea](https://mellea.ai) framework. For detailed examples on how to use this and other adapter, please refer to the [Mellea examples](https://github.com/generative-computing/mellea/tree/main/docs/examples/intrinsics).
+The recommended way to call this adapter is through the [Mellea](https://mellea.ai) framework. For detailed examples on how to use this and other adapters, please refer to the [Mellea examples](https://github.com/generative-computing/mellea/tree/main/docs/examples/intrinsics).
 
 
 ## Training Details
@@ -81,6 +81,7 @@ The training dataset consists of:
 - Human-generated underspecified questions created by annotators over content from Gov and ClapNQ domains
 - Multi-turn conversations with underspecified final turns
 - Synthetically generated clarification requests using target patterns (hedging with answers, hedging list, open domain)
+- Conversation-only examples (without document passages) targeting pre-retrieval scenarios
 
 The training dataset is proprietary and was obtained in combination with a third-party company who contracted the human annotators.
 
@@ -105,9 +106,9 @@ We evaluate the adapter on two dimensions: (1) classification accuracy in detect
 ### Test Set
 
 The evaluation uses a test set containing:
-- 80 positive examples: conversations where the last user question is underspecified
-- 150 random negatives: conversations where the last user question is clear
-- 50 hard negatives: conversations intended to be underspecified but adjudicated as clear by humans
+- 104 positive examples: conversations where the last user question is underspecified (generated by a combination of humans and AI model -- mistralai/Mistral-Large-3-675B-Instruct-2512)
+- 200 random negatives: randomly sampled, pre-existing conversations where the last user question is clear
+- 79 hard negatives: conversations where the last user question was intended to be underspecified (generated by an AI model -- mistralai/Mistral-Large-3-675B-Instruct-2512) but adjudicated as clear by humans
 
 ### Classification Results
 
@@ -115,39 +116,39 @@ The evaluation uses a test set containing:
 
 **Trained LoRA:**
 
-| Model | Underspecified (80) | CLEAR - random (150) | CLEAR - hard (50) | Overall Accuracy |
+| Model | Underspecified (104) | CLEAR - random (200) | CLEAR - hard (79) | Overall Accuracy |
 |-------|---------------------|----------------------|-------------------|------------------|
-| Granite-4.0-micro LoRA | 76.25% | 99.33% | 80% | **89.3%** |
+| Granite-4.0-micro LoRA | **84.6%** | **98.5%** | **87.3%** | **92.4%** |
 
 **OOB models (prompted, zero-shot):**
 
-| Model | Underspecified (80) | CLEAR - random (150) | CLEAR - hard (50) | Overall Accuracy |
+| Model | Underspecified (104) | CLEAR - random (200) | CLEAR - hard (79) | Overall Accuracy |
 |-------|---------------------|----------------------|-------------------|------------------|
-| gpt-oss-20b | 85% | 78% | 62% | 77.14% |
-| GPT 4o | 60% | 85.3% | 48% | 71.42% |
-| Granite-4.0-micro | 97.5% | 5.3% | 0% | 30.71% |
+| gpt-4o | 48.1% | 90.5% | 92.4% | 79.4% |
+| gpt-oss-20b | 90.4% | 72.5% | 58.2% | 74.4% |
+| Granite-4.0-micro | 25.0% | 85.0% | 63.3% | 64.2% |
 
-The LoRA achieves higher overall accuracy due to its ability to correctly identify clear queries, while prompted models tend to over-clarify.
+The LoRA achieves the highest overall accuracy by balancing high recall on underspecified queries with strong specificity on clear queries. Prompted models tend to err in one direction: gpt-oss-20b over-clarifies (high underspecified recall but low CLEAR accuracy), while gpt-4o and base Granite under-clarify (low underspecified recall).
 
 ### Clarification Quality Results
 
-For queries identified as underspecified, we measure the quality of generated clarification requests using an LLM judge (Llama-3.3-70b). The numbers in parentheses indicate how many of the 80 underspecified records each model correctly identified and generated clarifications for—quality metrics are measured only on those data points.
+For queries correctly identified as underspecified, we measure the quality of generated clarification requests using an LLM judge (Llama-3.3-70b). The judge compares the entities and options surfaced in the generated clarification against those in the gold-standard reference: precision measures what fraction of the generated options are correct, and recall measures what fraction of the gold options are covered. Additionally, the numbers in parentheses (near the model name) indicate how many of the 104 underspecified records each model correctly identified and generated clarifications for -- the quality metrics are measured only on those data points.
 
 **Trained LoRA:**
 
 | Model | Precision | Recall | F1 |
 |-------|-----------|--------|-----|
-| Granite-4.0-micro LoRA (61/80) | 90.4% | 80.1% | 84.9% |
+| Granite-4.0-micro LoRA (88/104) | **91.7%** | **87.5%** | **89.5%** |
 
 **OOB models (prompted, zero-shot):**
 
 | Model | Precision | Recall | F1 |
 |-------|-----------|--------|-----|
-| GPT 4o (48/80) | 91.9% | 83.4% | 87.4% |
-| gpt-oss-20b (68/80) | 86.9% | 66.9% | 75.6% |
-| Granite-4.0-micro (78/80) | 74.9% | 58.0% | 65.4% |
+| gpt-oss-20b (93/104) | 93.3% | 70.7% | 80.5% |
+| gpt-4o (50/104) | 81.3% | 69.8% | 75.1% |
+| Granite-4.0-micro (26/104) | 75.0% | 45.7% | 56.8% |
 
-The LoRA achieves clarification quality close to GPT-4o while maintaining much better classification accuracy and evaluating on more data points.
+The LoRA achieves the highest F1 (89.5%) across the largest number of correctly classified records (88/104), demonstrating both strong classification and high generation quality.
 
 
 ### Adapter Details
@@ -161,16 +162,16 @@ The LoRA achieves clarification quality close to GPT-4o while maintaining much b
 | **Target Modules** | q_proj, k_proj, v_proj, o_proj, input_linear, output_linear |
 
 **Infrastructure:**
-We trained the query clarification granite-4.0-micro LoRA adapter on IBM's Vela cluster using 8 H100 GPUs.
+We trained the query clarification granite-4.0-micro LoRA adapter on IBM's Vela cluster using 1 H100 GPU.
 
 **Ethical Considerations & Limitations:**
 The model's outputs are not guaranteed to be factually accurate or complete. All outputs should be independently validated before use in decision-making or downstream applications. The model has been trained and evaluated on English data only.
 
 ## Resources
 
-- ⭐️ Learn about the latest updates with Granite: https://www.ibm.com/granite
-- 📄 Get started with tutorials, best practices, and prompt engineering advice: https://www.ibm.com/granite/docs/
-- 💡 Learn about the latest Granite learning resources: https://github.com/ibm-granite/granite-guardian/tree/main/cookbooks
+- Learn about the latest updates with Granite: https://www.ibm.com/granite
+- Get started with tutorials, best practices, and prompt engineering advice: https://www.ibm.com/granite/docs/
+- Learn about the latest Granite learning resources: https://github.com/ibm-granite/granite-guardian/tree/main/cookbooks
 
 ## Contact
-[Lucian Popa](mailto:lpopa@us.ibm.com)
+[Lucian Popa](mailto:lpopa@us.ibm.com)