Intrinsics for Answerability Classification
Model Summary
This is a RAG-specific family of intrinsics fine-tuned for binary answerability classification task. The model takes as input a multi-turn conversation and a set of documents, and classifies whether the user's final query is answerable or unanswerable based on the available information in the documents.
We provide two intrinsics implemented as LoRA adapters (LoRA/aLoRA) trained over Granite-3.3-2b-instruct, Granite-3.3-8b-instruct, and GPT-OSS 20b.
- Developer: IBM Research
- Model type: LoRA and aLoRA adapter for ibm-granite/granite-3.3-2b-instruct, ibm-granite/granite-3.3-8b-instruct, and openai/gpt-oss-20b
- License: Apache 2.0
Intended use
This is a family of intrinsincs that enables answerability classification for the final user query in a multi-turn conversation, with respect to a set of provided documents. The model is trained to determine whether the last user query is answerable or unanswerable, based solely on the information present in the documents. This makes it suitable for applications involving RAG and document-grounded chatbots, where knowing whether sufficient information exists to answer a query is crucial. The classification output from the answerability model can be used in several downstream applications, including but not limited to:
- Filter out unanswerable questions before sending them to generation in RAG setting. By classifying a query as unanswerable upfront, the system can prevent hallucinated or misleading responses.
- Re-query the retriever to get more relevant documents. If a query is initially deemed unanswerable, the retriever can be re-invoked with alternate formulations to fetch more relevant documents.
Model input: The input to the answerability intrinsic is an
OpenAI-compatible chat completion request, containing a list of conversation
turns that can alternate between the user and assistant role and ending with
a user turn, as well as list of documents.
Model output: The output of the answerability intrinsic is the result of the original chat completion request formatted as a JSON object containing the answerability likelihood score.
Please see the code snippets in the Quickstart Example section below for examples that illustrate the intrinsic's input/output.
Quickstart Example
The recommended way to call this intrinsic is through the Mellea framework. Here is some example code for calling this intrinsic from Mellea:
from mellea.backends.huggingface import LocalHFBackend
from mellea.stdlib.base import ChatContext, Document
from mellea.stdlib.chat import Message
from mellea.stdlib.intrinsics import rag
backend = LocalHFBackend(model_id="ibm-granite/granite-3.3-2b-instruct")
context = ChatContext().add(Message("assistant", "Hello there, how can I help you?"))
next_user_turn = "What is the square root of 4?"
documents_answerable = [Document("The square root of 4 is 2.")]
documents_unanswerable = [Document("The square root of 8 is not 2.")]
result = rag.check_answerability(next_user_turn, documents_answerable, context, backend)
print(f"Result of answerability check when answer is in documents: {result}")
result = rag.check_answerability(
next_user_turn, documents_unanswerable, context, backend
)
print(f"Result of answerability check when answer is not in documents: {result}")
Training Details
Training Data
The training data uses the publicly available Government corpus from MT-RAG as the source of documents. Based on this corpus, we constructed a dataset consisting of a mix of human-created and synthetically generated multi-turn conversations. It includes two types of examples: (1) Answerable queries, where the final user question can be answered based on the provided documents. These examples teach the adapter to recognize when sufficient information is present to support an answer. (2) Unanswerable queries, where the documents lack the necessary information to answer the final user query. We used Mixtral as an automatic judge to validate the answerability labels and filter out noisy samples.
Training Hyperparameters
The LoRA adapter was fine-tuned using PEFT under the following regime: rank = 32, learning rate = 5e-6, number of epochs = 25, with early stopping based on validation set, and 90/10 split between training and validation.
Evaluation
Answerability Classification
We evaluated the model on binary answerability classification using MT-RAG Benchmark. In this setting, the model is given the full multi-turn conversation history along with the supporting documents. This benchmark evaluates the model's ability to assess answerability when the final user query can also depend on prior turns for context. The following table presents results comparing baselines and frontier models with task-specific answerability intrinsics on the answerability classification task on MT-RAG data. The LoRAs consistently outperform frontier models, converging near ~90% accuracy regardless of base model size. Even small models like Granite 3.3-2B, once fine-tuned, match or surpass much larger models, including GPT-4o. The difference between LoRA and aLoRA is minimal, indicating both are effective fine-tuning strategies.
| Models | Unanswerable F1 | Answerable F1 | Classification Accuracy | Weighted F1 | |
|---|---|---|---|---|---|
| Baselines | BigBird (pre-trained embeddings) w/ MLP | 73.4 | 65.2 | 69.8 | 69.6 |
| llama2-7b as classifier (Full SFT) | 88.2 | 85.9 | 87.1 | 87.1 | |
| Frontier Models out-of-the-box | Granite 3.3-2b-instruct | 48.7 | 70.4 | 62.4 | 58.7 |
| Granite 3.3-8b-instruct | 62.8 | 65.2 | 64.5 | 63.9 | |
| GPT-OSS-20b | 77.3 | 58.3 | 70.7 | 68.5 | |
| GPT-OSS-120b | 70.2 | 68.9 | 69.8 | 69.6 | |
| GPT4o-mini | 82.7 | 78.1 | 80.8 | 80.6 | |
| GPT4o | 85.7 | 77.5 | 82.5 | 81.9 | |
| Trained LoRAs/aLoRAs | Granite 3.3-2b LoRA | 91.2 | 89.6 | 90.4 | 90.5 |
| Granite 3.3-8b LoRA | 91.1 | 90.3 | 90.6 | 90.7 | |
| GPT-OSS-20b LoRA | 91.6 | 89.8 | 90.8 | 90.8 | |
| Granite 3.3-2b aLoRA | 89.8 | 88.6 | 89.1 | 89.2 | |
| Granite 3.3-8b aLoRA | 90.1 | 89.6 | 89.5 | 89.9 | |
| GPT-OSS-20b aLoRA | 90.4 | 88.6 | 89.6 | 89.6 |
Comparing the Answerability Intrinsics vs. Vanilla Granite Models for Answer Quality
We compare the performance of Granite 3.3-2b, Granite 3.3-8b Instruct vs. answerability intrinsics implemented as LoRA adapters on a subset of MT-RAG Benchmark. In this setup, each query is paired with only 5 retrieved passages as context.
Answerability Classification Performance: The answerability intrinsics outperform the vanilla model in overall F1 on both answerables and unanswerables. The answerability intrinsics achieves higher recall on unanswerable queries, making it better at identifying questions that should not be answered. However, this comes at the cost of lower recall on answerable queries.
Joint Answerability-Faithfulness Score computed as: > = 1 (if model prediction = IDK/unanswerable ∩ ground truth = unanswerable)
= RAGAS Faithfulness (if model prediction = non-IDK/answerable ∩ ground truth = answerable)
= 0 (otherwise)
This score rewards the model for correctly abstaining on unanswerable queries (full credit) and for providing faithful answers on answerable queries (partial credit based on RAGAS Faithfulness). No credit is given for incorrect or unfaithful predictions.
The answerability intrinsics for granite-2b and granite-8b achieves 8% and 13% lifts on this metric respectively. This rewards the model for correctly abstaining on unanswerable queries and for being faithful when it chooses to answer.
| F1 Score Unanswerable | F1 Score Answerable | Recall Unanswerable | Recall Answerable | Joint Answerability- Faithfulness Score | |
|---|---|---|---|---|---|
| Granite 3.3-2b Instruct | 13 | 77 | 7 | 99 | 48 |
| Granite 3.3-2b LoRA | 48 | 78 | 37 | 89 | 56 |
| Granite 3.3-8b Instruct | 17 | 77 | 10 | 99 | 49 |
| Granite 3.3-8b LoRA | 65 | 81 | 60 | 86 | 62 |
Model Card Authors
Framework versions
- PEFT 0.14.0