Query clarification migration (#9)

- feat: add query clarification intrinsic, migrate from old repo (848be843a39ccbf34969550c63a8299782305e40)
- feat: add query clarification model for Ollama (60df3556fda54957cdeb422121858ab9c91f99b7)
- feat: add query_clarification IO yaml file (a288529f5283d3923bd6b973475133abaaf32b04)

query_clarification/README.md

@@ -0,0 +1,160 @@
+---
+license: apache-2.0
+language:
+- en
+pipeline_tag: text-generation
+library_name: peft
+library_name: transformers
+---
+---
+
+# Query Clarification Intrinsic
+
+## Model Summary
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+We are releasing a query clarification intrinsic, implemented as a family of adapters that are fine-tuned specifically for the following task:
+
+    Given a user query in a multi-turn conversation context, generate clarification questions
+    that help disambiguate and better understand the user's intent when the query is ambiguous
+    or incomplete. The clarification questions should help refine the query to be more specific
+    and actionable.
+
+While the intrinsic is general purpose, one of its main use cases is in RAG settings where generating clarifying questions can help retrieve more relevant documents and provide better answers. Outside of RAG, there are other conversational use cases that require query clarification, for example, in customer support scenarios, database query interfaces, or multi-step reasoning systems. The intrinsic uses only the current query and conversation context to generate targeted clarification questions. We are providing experimental results in a RAG setting as well as on a specialized enterprise setup, showing in both cases that the intrinsic performance is significantly higher than when prompting out-of-the-box models, including open-source models and frontier models.
+
+The adapters released here work with the IBM granite-4.0-micro model.
+
+- **Developer:** IBM Research
+- **Model type:** LoRA adapter for [ibm-granite/granite-4.0-micro](https://huggingface.co/ibm-granite/granite-4.0-micro)
+- **License:** [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0)
+
+
+## Intended use
+
+The intrinsic gives the ability to generate clarification questions for a user query in a multi-turn conversation. These clarification questions are generated when the user's query may be ambiguous, incomplete, or when additional context is needed to better understand the user's intent. The generated questions can help guide the user to provide more specific information or can be used internally to better understand which aspect of the query to focus on.
+
+The clarification questions can be useful in several scenarios:
+- Helping users refine their queries before retrieval in a RAG setting
+- Supporting interactive query refinement workflows
+- Improving answer quality by identifying ambiguities early
+- Enhancing user experience in conversational AI systems
+
+> [!TIP]
+> Note: While you can invoke the query clarification intrinsic directly, it is strongly recommended to call it through [granite-common](https://github.com/ibm-granite/granite-common), which wraps the model with a tailored I/O processor, enabling a friendlier development interface. The I/O processor takes care of several data transformation and validation tasks that would be otherwise required (including sending the appropriate instruction for the intrinsic) as well as validating the intrinsic's output. We next describe the input/output of the query clarification intrinsic when invoked through granite-common.
+
+**Intrinsic input**: The input to the query clarification intrinsic is an OpenAI-compatible chat completion request, containing a list of conversation turns that can alternate between the `user` and `assistant` role, and ending with a `user` turn. The last `user` turn in the list is assumed to be the query that may need clarification.
+
+**Intrinsic output**: The output of the query clarification intrinsic is the result of the original chat completion request formatted as a
+JSON object as follows:
+```json
+{
+  "clarification": <Clarification question(s) to help disambiguate the user's intent>
+}
+```
+
+Please see the code snippets in the Quickstart Example section below for examples that illustrate the intrinsic's input/output.
+
+## Quickstart Example
+
+The recommended way to call this intrinsic is through the [Mellea](https://mellea.ai) framework.
+Here is some example code for calling this intrinsic from Mellea:
+```
+from mellea.backends.huggingface import LocalHFBackend
+from mellea.stdlib.base import ChatContext
+from mellea.stdlib.chat import Message
+from mellea.stdlib.intrinsics import rag
+
+
+backend = LocalHFBackend(model_id="ibm-granite/granite-4.0-micro")
+context = (
+    ChatContext()
+    .add(Message("assistant", "How can I help you today?"))
+    .add(Message("user", "I want to learn about cats."))
+    .add(
+        Message(
+            "assistant",
+            "I'd be happy to help! There's a lot to cover about cats. "
+            "Are you interested in cat behavior, care, history, or something else?",
+        )
+    )
+)
+next_user_turn = "Tell me about them."
+
+print(f"Original user question: {next_user_turn}")
+
+result = rag.clarify_question(next_user_turn, context, backend)
+print(f"Clarification: {result}")
+```
+
+
+## Training Details
+
+The training data contains examples with varying degrees of query ambiguity and incompleteness. The intrinsic learns to recognize when a query lacks sufficient specificity and to generate targeted clarification questions that address the identified ambiguities.
+
+### Training Data
+
+The training data is derived from conversational datasets and RAG scenarios where query ambiguity is common. The dataset includes examples with:
+- Ambiguous references or underspecified entities
+- Missing context or implicit assumptions
+- Multiple possible interpretations of the user's intent
+- Queries that could benefit from additional information
+
+These examples are paired with high-quality clarification questions that effectively disambiguate the queries. Additionally, we have used synthetically generated examples to maximize coverage of different ambiguity patterns and improve the intrinsic's ability to handle diverse query types.
+
+#### Model Variants
+
+The intrinsic is available as a LoRA adapter for the following base models:
+- IBM Granite 4.0 Micro
+
+#### System Prompt Variations
+
+During training, we tested the adapter with multiple system prompt variations to ensure robustness:
+
+Version 1: <|start_of_role|>system<|end_of_role|> You are a helpful assistant that generates clarification questions to better understand ambiguous or incomplete user queries.<|end_of_text|>
+
+Version 2: <|start_of_role|>system<|end_of_role|> Knowledge Cutoff Date: April 2024. Today's Date: May 20, 2025. You are Granite, developed by IBM. Generate clarification questions that help disambiguate the user's intent based on the conversation context.<|end_of_text|>
+
+Version 3: An empty system prompt (no instructions provided).
+
+This approach ensures that the adapters remain effective and reliable across varying system prompt formats commonly encountered in real-world RAG applications.
+
+#### Training Hyperparameters
+
+The adapter was fine-tuned using PEFT under the following regime: rank = 32, learning rate = 3e-6, number of epochs = 25, and linear learning rate scheduler.
+
+
+## Evaluation
+
+We provide two types of evaluation: 1) evaluation specific to the task of query clarification, 2) end-to-end evaluation in a RAG setting, where we evaluate how query clarification impacts the quality of retrieved documents and final answer generation.
+
+### Evaluation Specific for the Query Clarification Task
+
+Here we evaluate the quality of the generated clarification questions on an enterprise internal dataset. This dataset contains queries with varying degrees of ambiguity, paired with human-validated clarification questions. We evaluate the model-generated clarifications using both automatic metrics and an LLM judge (with Llama-3.3-70b as the model) with specific prompts to assess whether the generated questions effectively disambiguate the original query.
+
+The evaluation focuses on:
+- Relevance: Do the clarification questions address actual ambiguities in the query?
+- Quality: Are the questions clear, concise, and helpful?
+- Efficiency: Do the questions effectively narrow down the user's intent with minimal overhead?
+
+### End-to-End Evaluation in RAG Settings
+
+We evaluate the query clarification intrinsic in a complete RAG pipeline where:
+1. A user provides an initial query
+2. Clarification questions are generated
+3. The clarified query (or queries) are used for retrieval
+4. The retrieved documents are used for answer generation
+
+We measure improvements in:
+- Retrieval relevance and recall metrics
+- Final answer quality and relevance to the user's actual intent
+- User satisfaction in interactive query refinement scenarios
+
+
+## Contact
+
+[Lucian Popa](mailto:lpopa@us.ibm.com)
+
+### Framework versions
+
+- PEFT 0.17.2

query_clarification/granite-4.0-micro/lora/adapter_config.json

@@ -0,0 +1,44 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "ibm-granite/granite-4.0-micro",
+  "bias": "none",
+  "corda_config": null,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 64,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.17.2.dev0@ec5a1b2ce62b14925239f62582eb3db5d9174af2",
+  "qalora_group_size": 16,
+  "r": 32,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "input_linear",
+    "output_linear",
+    "k_proj",
+    "q_proj",
+    "o_proj",
+    "v_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

query_clarification/granite-4.0-micro/lora/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8bf22d4fe51f403feac79cc081fa1297e365591b48e5cd48b782153a0d054cf5
+size 235995984

query_clarification/granite-4.0-micro/lora/io.yaml

@@ -0,0 +1,22 @@
+# Model name string, or null to use whatever is provided in the chat completion request
+model: ~
+# JSON schema of the model's output
+response_format: |
+  {
+    "properties": {
+      "clarification": {
+        "title": "clarification",
+        "type": "string"
+      }
+    },
+    "required": [
+      "clarification"
+    ],
+    "title": "QueryClarificationOutput",
+    "type": "object"
+  }
+transformations: ~
+instruction: ~
+parameters:
+  max_completion_tokens: 1024
+sentence_boundaries: false

query_clarification/granite4_micro/lora/Lora-q8_0.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0715cd4409af83a965f873df34351e0ec1b8aedf347926ec8f259728dd454624
+size 288

query_clarification/granite4_micro/lora/Modelfile

@@ -0,0 +1,2 @@
+FROM granite4:micro
+ADAPTER Lora-q8_0.gguf

query_clarification/granite4_micro/lora/io.yaml

@@ -0,0 +1,21 @@
+model: null
+response_format: null
+transformations: null
+instruction: null
+parameters:
+  response_format:
+    type: json_schema
+    json_schema:
+      schema:
+        properties:
+          clarification:
+            title: clarification
+            type: string
+        required:
+        - clarification
+        title: QueryClarificationOutput
+        type: object
+      strict: true
+  max_tokens: 1024
+sentence_boundaries: false
+docs_as_message: roles