Update query_clarification/README.md (#13)

- Update query_clarification/README.md (a2a4f3b8f4429e4c0da1588d859b8b27a7b1b666)


Co-authored-by: Lucian Popa <lucianpopa@users.noreply.huggingface.co>

query_clarification/README.md

@@ -6,24 +6,20 @@ pipeline_tag: text-generation
 library_name: peft
 library_name: transformers
 ---
----
 
 # Query Clarification Intrinsic
 
 ## Model Summary
 
-<!-- Provide a quick summary of what the model is/does. -->
-
-We are releasing a query clarification intrinsic, implemented as a family of adapters that are fine-tuned specifically for the following task:
+We are releasing a query clarification intrinsic, implemented as a LoRA adapter fine-tuned specifically for the following task:
 
-    Given a user query in a multi-turn conversation context, generate clarification questions
-    that help disambiguate and better understand the user's intent when the query is ambiguous
-    or incomplete. The clarification questions should help refine the query to be more specific
-    and actionable.
+    Given a multi-turn conversation between a user and an AI assistant (and optionally relevant content such as
+    RAG documents), detect whether the last user query is underspecified (no clear interpretation or multiple
+    valid interpretations) and, if so, formulate an appropriate clarification request back to the user.
 
-While the intrinsic is general purpose, one of its main use cases is in RAG settings where generating clarifying questions can help retrieve more relevant documents and provide better answers. Outside of RAG, there are other conversational use cases that require query clarification, for example, in customer support scenarios, database query interfaces, or multi-step reasoning systems. The intrinsic uses only the current query and conversation context to generate targeted clarification questions. We are providing experimental results in a RAG setting as well as on a specialized enterprise setup, showing in both cases that the intrinsic performance is significantly higher than when prompting out-of-the-box models, including open-source models and frontier models.
+The intrinsic is designed for conversational use cases where user queries may be ill-formed, unclear, or have multiple valid interpretations based on the underlying system or content. Use cases include RAG systems, routing, and tool invocation scenarios. We provide experimental results showing that the intrinsic achieves significantly higher classification accuracy than prompted out-of-the-box models, including frontier models such as gpt-4o.
 
-The adapters released here work with the IBM granite-4.0-micro model.
+The adapter released here works with the IBM granite-4.0-micro model.
 
 - **Developer:** IBM Research
 - **Model type:** LoRA adapter for [ibm-granite/granite-4.0-micro](https://huggingface.co/ibm-granite/granite-4.0-micro)
@@ -32,129 +28,126 @@ The adapters released here work with the IBM granite-4.0-micro model.
 
 ## Intended use
 
-The intrinsic gives the ability to generate clarification questions for a user query in a multi-turn conversation. These clarification questions are generated when the user's query may be ambiguous, incomplete, or when additional context is needed to better understand the user's intent. The generated questions can help guide the user to provide more specific information or can be used internally to better understand which aspect of the query to focus on.
+The intrinsic detects whether the last user query in a multi-turn conversation is underspecified and generates a clarification request if needed. A query may be underspecified due to:
 
-The clarification questions can be useful in several scenarios:
-- Helping users refine their queries before retrieval in a RAG setting
-- Supporting interactive query refinement workflows
-- Improving answer quality by identifying ambiguities early
-- Enhancing user experience in conversational AI systems
+- **Ambiguous references**: The query refers to entities or concepts that have multiple valid interpretations in the content (e.g., "What browser are you using?" when multiple browsers are documented)
+- **Ill-formed questions**: The query is unclear or makes implicit references that cannot be resolved from context alone
+- **Multiple valid choices**: The user's intent could map to several different actions, routes, or entities in the system
 
-> [!TIP]
-> Note: While you can invoke the query clarification intrinsic directly, it is strongly recommended to call it through [granite-common](https://github.com/ibm-granite/granite-common), which wraps the model with a tailored I/O processor, enabling a friendlier development interface. The I/O processor takes care of several data transformation and validation tasks that would be otherwise required (including sending the appropriate instruction for the intrinsic) as well as validating the intrinsic's output. We next describe the input/output of the query clarification intrinsic when invoked through granite-common.
+The intrinsic generates clarification requests using different patterns depending on the number of matching options:
+- **Hedging with answers**: For a small number of options, provide partial answers for each interpretation
+- **Hedging list**: For a medium number of options, list the possible interpretations and ask which one
+- **Open domain**: For many possible matches, ask an open-ended clarifying question
 
-**Intrinsic input**: The input to the query clarification intrinsic is an OpenAI-compatible chat completion request, containing a list of conversation turns that can alternate between the `user` and `assistant` role, and ending with a `user` turn. The last `user` turn in the list is assumed to be the query that may need clarification.
+**Intrinsic input**: The input is an OpenAI-compatible chat completion request containing conversation turns (alternating `user` and `assistant` roles) ending with a `user` turn, optionally with relevant content/documents.
 
-**Intrinsic output**: The output of the query clarification intrinsic is the result of the original chat completion request formatted as a
-JSON object as follows:
+**Intrinsic output**: The output is a JSON object as follows:
 ```json
 {
-  "clarification": <Clarification question(s) to help disambiguate the user's intent>
+  "clarification": <Clarification question or "CLEAR">
 }
 ```
+The `clarification` field contains either a clarification question if the query is underspecified, or `CLEAR` if the query does not need clarification.
 
-Please see the code snippets in the Quickstart Example section below for examples that illustrate the intrinsic's input/output.
+### Example
 
-## Quickstart Example
+**Input conversation:**
 
-The recommended way to call this intrinsic is through the [Mellea](https://mellea.ai) framework.
-Here is some example code for calling this intrinsic from Mellea:
-```
-from mellea.backends.huggingface import LocalHFBackend
-from mellea.stdlib.base import ChatContext
-from mellea.stdlib.chat import Message
-from mellea.stdlib.intrinsics import rag
-
-
-backend = LocalHFBackend(model_id="ibm-granite/granite-4.0-micro")
-context = (
-    ChatContext()
-    .add(Message("assistant", "How can I help you today?"))
-    .add(Message("user", "I want to learn about cats."))
-    .add(
-        Message(
-            "assistant",
-            "I'd be happy to help! There's a lot to cover about cats. "
-            "Are you interested in cat behavior, care, history, or something else?",
-        )
-    )
-)
-next_user_turn = "Tell me about them."
-
-print(f"Original user question: {next_user_turn}")
-
-result = rag.clarify_question(next_user_turn, context, backend)
-print(f"Clarification: {result}")
+| Role | Message |
+|------|---------|
+| user | Is the language that descended from Common Brittonic still spoken today, or has it become extinct? |
+
+**Relevant content (RAG documents):** Information about Celtic Britons, Common Brittonic language, Welsh, Breton, Cornish, and Cumbric languages.
+
+**Output:**
+```json
+{
+  "clarification": "Several languages descended from Common Brittonic are still spoken or have been revived, like Welsh—a living language in Wales—Breton, spoken in Brittany, or Cornish, which was extinct but has seen recent revitalization. Which one are you referring to?"
+}
 ```
 
+In this example, the intrinsic detects that the user's question is underspecified because multiple languages descended from Common Brittonic, and generates a clarification request using the "hedging with answers" pattern—providing partial answers for each interpretation before asking for clarification.
+
+## Usage Examples
+
+The recommended way to call this intrinsic is through the [Mellea](https://mellea.ai) framework. For detailed examples on how to use this and other intrinsics, please refer to the [Mellea intrinsics examples](https://github.com/generative-computing/mellea/tree/main/docs/examples/intrinsics).
+
 
 ## Training Details
 
-The training data contains examples with varying degrees of query ambiguity and incompleteness. The intrinsic learns to recognize when a query lacks sufficient specificity and to generate targeted clarification questions that address the identified ambiguities.
+The training data contains both positive examples (underspecified queries with corresponding clarification requests) and negative examples (clear queries that should not trigger clarification). Hard negatives are included to help the model learn the boundary between clear and underspecified queries.
 
 ### Training Data
 
-The training data is derived from conversational datasets and RAG scenarios where query ambiguity is common. The dataset includes examples with:
-- Ambiguous references or underspecified entities
-- Missing context or implicit assumptions
-- Multiple possible interpretations of the user's intent
-- Queries that could benefit from additional information
-
-These examples are paired with high-quality clarification questions that effectively disambiguate the queries. Additionally, we have used synthetically generated examples to maximize coverage of different ambiguity patterns and improve the intrinsic's ability to handle diverse query types.
+The training dataset consists of:
+- Human-generated underspecified questions created by annotators over content from Gov and ClapNQ domains
+- Multi-turn conversations with underspecified final turns
+- Synthetically generated clarification requests using target patterns (hedging with answers, hedging list, open domain)
 
-#### Model Variants
+The training dataset is proprietary and was obtained in combination with a third-party company who contracted the human annotators.
 
-The intrinsic is available as a LoRA adapter for the following base models:
-- IBM Granite 4.0 Micro
+### Robustness to System Prompts
 
-#### System Prompt Variations
+Different researchers or practitioners may use various system prompts tailored to their specific use cases. For the IBM granite versions of the adapter, to enhance robustness against these variations, we generated multiple versions of each training sample with different system prompts.
 
-During training, we tested the adapter with multiple system prompt variations to ensure robustness:
+#### Training Hyperparameters
 
-Version 1: <|start_of_role|>system<|end_of_role|> You are a helpful assistant that generates clarification questions to better understand ambiguous or incomplete user queries.<|end_of_text|>
+The adapter was fine-tuned using PEFT under the following regime:
+- Learning rate: 1e-5
+- Rank: 32
+- LoRA alpha: 64
+- Dropout: 0.05
+- Epochs: 20 with early stopping
+- Hard negatives ratio: 0.7
 
-Version 2: <|start_of_role|>system<|end_of_role|> Knowledge Cutoff Date: April 2024. Today's Date: May 20, 2025. You are Granite, developed by IBM. Generate clarification questions that help disambiguate the user's intent based on the conversation context.<|end_of_text|>
 
-Version 3: An empty system prompt (no instructions provided).
+## Evaluation
 
-This approach ensures that the adapters remain effective and reliable across varying system prompt formats commonly encountered in real-world RAG applications.
+We evaluate the intrinsic on two dimensions: (1) classification accuracy in detecting underspecified vs. clear queries, and (2) quality of generated clarification requests.
 
-#### Training Hyperparameters
+### Test Set
 
-The adapter was fine-tuned using PEFT under the following regime: rank = 32, learning rate = 3e-6, number of epochs = 25, and linear learning rate scheduler.
+The evaluation uses a test set containing:
+- 80 positive examples: conversations where the last user question is underspecified
+- 150 random negatives: conversations where the last user question is clear
+- 50 hard negatives: conversations intended to be underspecified but adjudicated as clear by humans
 
+### Classification Results
 
-## Evaluation
+**Overall Accuracy:**
 
-We provide two types of evaluation: 1) evaluation specific to the task of query clarification, 2) end-to-end evaluation in a RAG setting, where we evaluate how query clarification impacts the quality of retrieved documents and final answer generation.
+**Trained LoRA:**
 
-### Evaluation Specific for the Query Clarification Task
+| Model | Underspecified (80) | CLEAR - random (150) | CLEAR - hard (50) | Overall Accuracy |
+|-------|---------------------|----------------------|-------------------|------------------|
+| Granite-4.0-micro LoRA | 81.25% | 100% | 86% | **92.14%** |
 
-Here we evaluate the quality of the generated clarification questions on an enterprise internal dataset. This dataset contains queries with varying degrees of ambiguity, paired with human-validated clarification questions. We evaluate the model-generated clarifications using both automatic metrics and an LLM judge (with Llama-3.3-70b as the model) with specific prompts to assess whether the generated questions effectively disambiguate the original query.
+**OOB models (prompted, zero-shot):**
 
-The evaluation focuses on:
-- Relevance: Do the clarification questions address actual ambiguities in the query?
-- Quality: Are the questions clear, concise, and helpful?
-- Efficiency: Do the questions effectively narrow down the user's intent with minimal overhead?
+| Model | Underspecified (80) | CLEAR - random (150) | CLEAR - hard (50) | Overall Accuracy |
+|-------|---------------------|----------------------|-------------------|------------------|
+| gpt-oss-20b | 85% | 78% | 62% | 77.14% |
+| GPT 4o | 60% | 85.3% | 48% | 71.42% |
+| Granite-4.0-micro | 97.5% | 5.3% | 0% | 30.71% |
 
-### End-to-End Evaluation in RAG Settings
+The LoRA achieves significantly higher overall accuracy due to its ability to correctly identify clear queries, while prompted models tend to over-clarify.
 
-We evaluate the query clarification intrinsic in a complete RAG pipeline where:
-1. A user provides an initial query
-2. Clarification questions are generated
-3. The clarified query (or queries) are used for retrieval
-4. The retrieved documents are used for answer generation
+### Clarification Quality Results
 
-We measure improvements in:
-- Retrieval relevance and recall metrics
-- Final answer quality and relevance to the user's actual intent
-- User satisfaction in interactive query refinement scenarios
+For queries correctly identified as underspecified, we measure the quality of generated clarification requests using an LLM judge (Llama-3.3-70b). The numbers in parentheses indicate how many of the 80 underspecified records each model correctly identified and generated clarifications for—quality metrics are measured only on those data points.
 
+**Trained LoRA:**
 
-## Contact
+| Model | Precision | Recall | F1 |
+|-------|-----------|--------|-----|
+| Granite-4.0-micro LoRA (65/80) | 88.4% | 78.3% | 83.0% |
 
-[Lucian Popa](mailto:lpopa@us.ibm.com)
+**OOB models (prompted, zero-shot):**
 
-### Framework versions
+| Model | Precision | Recall | F1 |
+|-------|-----------|--------|-----|
+| GPT 4o (48/80) | 91.9% | 83.4% | 87.4% |
+| gpt-oss-20b (68/80) | 86.9% | 66.9% | 75.6% |
+| Granite-4.0-micro (78/80) | 74.9% | 58.0% | 65.4% |
 
-- PEFT 0.17.2
+The LoRA achieves clarification quality close to GPT-4o while maintaining much better classification accuracy and evaluating on more data points.