Update context relevance model card with granite 4 micro lora eval (#14)

- Update context relevance model card with granite 4 micro lora eval (664b90c175b03e67608efd3c82ac0afa75d155f6)


Co-authored-by: Li <Hong23@users.noreply.huggingface.co>

context_relevance/README.md

@@ -11,20 +11,20 @@ library_name: transformers
 
 ## Model Summary
 
-This is a RAG-specific family of intrinsics fine-tuned for the context relevancy task:
+This is a RAG-specific family of intrinsics fine-tuned for the context relevance task:
 
     Given (1) a document and (2) a multi-turn conversation between a user and an AI assistant, identify whether the document is relevant (including partially relevant) and useful to answering the last user question.
 While this adapter is general purpose, it is especially effective in RAG settings right after the retrieval model's step where the adapter can be used to identify documents or passages that may mislead or harm the downstream generator model's response generation.
 
 - **Developer:** IBM Research
-- **Model type:** LoRA and aLoRA adapter for [ibm-granite/granite-3.3-8b-instruct](https://huggingface.co/ibm-granite/granite-3.3-8b-instruct)
+- **Model type:** LoRA adapter for [ibm-granite/granite-4.0-micro](https://huggingface.co/ibm-granite/granitelib-rag-r1.0/tree/main/context_relevance/granite-4.0-micro)
   and [openai/gpt-oss-20b](https://huggingface.co/openai/gpt-oss-20b)
 - **License:** [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0)
 
 ## Intended use
 This is a family of intrinsincs that enables classification of relevant/irrelevant documents for the final user query in a multi-turn conversation. The model has been trained to determine whether a document is useful and relevant for answering the last user question in a multi-turn conversation.
 
-The classification output from the context relevancy model can be used in several downstream applications, including but not limited to:
+The classification output from the context relevance model can be used in several downstream applications, including but not limited to:
 
 - Filter out irrelevant and misleading documents before sending them to a generator model in a RAG setting.  By removing irrelevant documents upfront, the downsteam generator model reduces the likelihood of outputing misleading responses. Moreover, removing irrelevant documents or passages can help reduce eat up downstream context-length limits , thereby improving inference time.
 
@@ -35,7 +35,7 @@ The classification output from the context relevancy model can be used in severa
 1. A conversation formatted using the chat template
 2. The final user query extracted from the conversation
 3. A document to evaluate for relevance
-4. A special context relevancy invocation prompt
+4. A special context relevance invocation prompt
 
 The model uses a specific format with separate roles for each component:
 - Conversation: Applied via `tokenizer.apply_chat_template()`
@@ -61,34 +61,8 @@ The model uses a specific format with separate roles for each component:
 }
 ```
 
-## Quickstart Example
-
-The recommended way to call this intrinsic is through the [Mellea](https://mellea.ai) framework.
-Here is some example code for calling this intrinsic from Mellea:
-```
-from mellea.backends.huggingface import LocalHFBackend
-from mellea.stdlib.base import ChatContext, Document
-from mellea.stdlib.chat import Message
-from mellea.stdlib.intrinsics import rag
-
-
-backend = LocalHFBackend(model_id="ibm-granite/granite-3.3-2b-instruct")
-context = ChatContext()
-question = "Who is the CEO of Microsoft?"
-document = Document(
-    # Document text does not say who is the CEO.
-    "Microsoft Corporation is an American multinational corporation and technology "
-    "conglomerate headquartered in Redmond, Washington.[2] Founded in 1975, the "
-    "company became influential in the rise of personal computers through software "
-    "like Windows, and the company has since expanded to Internet services, cloud "
-    "computing, video gaming and other fields. Microsoft is the largest software "
-    "maker, one of the most valuable public U.S. companies,[a] and one of the most "
-    "valuable brands globally."
-)
-
-result = rag.check_context_relevance(question, document, context, backend)
-print(f"Result of context relevance check: {result}")
-```
+**Usage Example**:
+The recommended way to call this intrinsic is through the [Mellea](https://mellea.ai) framework, following the examples [here](https://github.com/generative-computing/mellea/tree/main/docs/examples/intrinsics).
 
 
 ## Training Details
@@ -106,7 +80,7 @@ The LoRA adapter was fine-tuned using PEFT under the following regime: rank = 32
 
 Our model was evaluated against several existing benchmarks to measure its ability to determine whether document is relevant or not to the last user question in a multi-turn conversation.
 
-### Forming evaluation datasets for context relevancy
+### Forming evaluation datasets for context relevance
 
 For each of the datasets, we created pairs of document and question pairs and generated a label (based on the available labels in the dataset) as to whether the document is relevant or useful to answering the question.
 
@@ -117,62 +91,41 @@ Given the dataset groundtruth and the LoRA outputs, we then calculated the preci
 
 ### Evaluation Datasets
 
-We evaluated on the following datasets from which we form documents and question pairs for an evaluation dataset (we also denote how we derive negative labels per dataset):
+We evaluated on the following five datasets:
 
-- Multi-turn conversational RAG benchmarks: MTRAG [MT-RAG](https://arxiv.org/pdf/2501.03468). The MT-RAG dataset comes with three settings of RAG retrieval: (1) Reference: perfect retriever (all passages are relevant to the question); (2) Reference + RAG: references passages + additional passages resulting in 5 total passages; (3) RAG: standard RAG setting (top 5 retrieved passages). We split the evaluation dataset according to the three settings and formed pairs of questions and passages.
-  - Pre-existing negative labels.
+- [CLAPNQ](https://arxiv.org/abs/2404.02103)
+- [RAG MINI BIOASQ](https://huggingface.co/datasets/enelpol/rag-mini-bioasq)
+- [Open Australian Legal QA](https://huggingface.co/datasets/isaacus/open-australian-legal-qa)
+- [FinanceBench](https://arxiv.org/abs/2311.11944)
+- [UCINLP DROP](https://aclanthology.org/N19-1246/)
 
-- SAGE RAG benchmarks used in evaluating Granite models: [FinanceBench](https://arxiv.org/abs/2311.11944), [BioASQ](https://huggingface.co/datasets/enelpol/rag-mini-bioasq), [Open Australian Legal Corpus](https://huggingface.co/datasets/isaacus/open-australian-legal-qa).
-  - Negative labels created by random pairings.
 
-- [Granite Guardian](https://arxiv.org/abs/2412.07724) datasets for context relevancy: HotpotQA and SquadV2.
-  - Pre-existing negative labels which were obtained via prompting a separate model over the original datasets.
-
-- Question Answering datasets, that [implicit-fact retrieval](https://arxiv.org/abs/2409.14924): [CLAPNQ](https://arxiv.org/abs/2404.02103) and [Drop](https://aclanthology.org/N19-1246/).
-  - Pre-existing negative labels.
+### Evaluation Results
 
-- [BEIR](https://arxiv.org/abs/2104.08663) benchmark datasets for information retrieval: [BioASQ](https://huggingface.co/datasets/enelpol/rag-mini-bioasq)
-  - Negative labels created by random pairings.
+We evaluated the LoRA adapters against various frontier models on the five evaluation datasets. Here, we create strong prompt instruction (three labels) for frontier models, but no instruction for LoRA adapters.
+The results show average F1 scores for relevant and irrelevant classifications, along with overall accuracy.
 
+| Model | Avg. F1 (Relevant) | Avg. F1 (Irrelevant) | Accuracy |
+|-------|-------------------|---------------------|----------|
+| **Frontier Models (Out of Box)** | | | |
+| Granite 3.3-8b Instruct | 94.9% | 92.3% | 94.7% |
+| Granite 4 micro | 93.8% | 88.6% | 93.1% |
+| GPT-OSS-20b | 90.1% | 85.7% | 90.0% |
+| GPT-OSS-120b | 93.7% | 91.4% | 93.6% |
+| GPT 4o mini | 93.7% | 90.8% | 93.6% |
+| GPT 4o | 95.6% | 93.9% | 95.7% |
+| GPT 5 mini | 95.8% | 93.9% | 95.6% |
+| **Trained LoRAs** | | | |
+| Granite 4 micro-context-relevance-LoRA | 93.2% | 87.1% | 92.1% |
+| Granite 4 micro-context-relevance-aLoRA | 87.0% | 72.8% | 83.4% |
 
-### Evaluation Results
+**Takeaways:**
+- LoRA without instruction is comparable to frontier models with strong instruction
 
-We evaluated the LoRA adapter against `meta-llama/Llama-3.3-70B-Instruct` and the base Granite model, `ibm-granite/granite-3.3-8b-instruct`.  For Llama, we prompted the model using the prompt [proven to perform best for context relevancy](https://arxiv.org/abs/2502.13908).
-
-#### Precision for relevant labels
-
-| Benchmark Type | Dataset  | meta-llama/Llama-3.3-70B-Instruct | ibm-granite/granite-guardian-3.1-5b | ibm-granite/granite-3.3-8b-instruct | Three-label LoRA | Delta from base |
-|-----------------------------|-----------------------------------------|-----------------------------------|-------------------------------------|-------------------------------------|----------------------------------------------------------|-----------------|
-| MTRAG | MTRAG (RAG) | 0.93 | 0.90 | 0.93 | 0.96 | 0.034 |
-| MTRAG | MTRAG Clean (RAG) (71 fixed) | 0.95 | 0.92 | 0.94 |   0.97 |  0.03 |
-| MTRAG | MTRAG (Reference + RAG) | 0.87 | 0.83 | 0.86 |   0.92 |  0.06 |
-| MTRAG | MTRAG (Reference)  | 1 | 1 | 1 |  1 | 0 |
-| QA, Implicit Fact Retrieval | PrimeQA/clapnq  | 0.97 | 0.31 | 0.69 |   0.99 |  0.30 |
-| QA, Implicit Fact Retrieval | ucinlp/drop | 0.86 | 0.35 | 0.73 |  0.94 | 0.21 |
-| Sage Benchmarks + BeIR | PatronusAI/financebench | 0.92 | 0.44 | 0.66 | 0.99 |  0.33 |
-| Sage Benchmarks + BeIR | enelpol/rag-mini-bioasq | 0.98 | 0.83 | 0.94 |   0.99 |  0.05 |
-| Sage Benchmarks + BeIR | isaacus/open-australian-legal-qa | 0.93 | 0.08 | 0.60 |  1 | 0.40 |
-| Granite Guardian  | HotpotQA + SquadV2  | 0.54 | 0.13 | 0.54 | 0.65 | 0.11 |
-
-
-#### Recall for irrelevant labels:
-
-| Benchmark Type | Dataset  | meta-llama/Llama-3.3-70B-Instruct | ibm-granite/granite-guardian-3.1-5b | ibm-granite/granite-3.3-8b-instruct | Three-label LoRA | Delta from base |
-|-----------------------------|-----------------------------------------|-----------------------------------|-------------------------------------|-------------------------------------|------------------|-----------------|
-|  MTRAG | MTRAG (RAG) | 0.41 | 0.21 | 0.29 | 0.73 | 0.44 |
-| MTRAG | MTRAG Clean (RAG) (71 fixed) | 0.44 | 0.21 | 0.33 | 0.74 | 0.41 |
-| MTRAG | MTRAG (Reference + RAG) | 0.39 | 0.21 | 0.29 | 0.73 | 0.44 |
-| MTRAG | MTRAG (Reference)  | 0 | 0 | 0 | 0 | 0 |
-| QA, Implicit Fact Retrieval | PrimeQA/clapnq  | 0.97 | 0 | 0.65 | 0.99 | 0.34 |
-| QA, Implicit Fact Retrieval | ucinlp/drop | 0.84 | 0.02 | 0.64 | 0.94 | 0.30 |
-| Sage Benchmarks + BeIR | PatronusAI/financebench | 0.89 | 0 | 0.34 | 0.99 | 0.65 |
-| Sage Benchmarks + BeIR | enelpol/rag-mini-bioasq | 0.87 | 0 | 0.53 | 0.99 | 0.46 |
-| Sage Benchmarks + BeIR | isaacus/open-australian-legal-qa | 0.92 | 0 | 0.36 |  1.0 | 0.64 |
-| Granite Guardian  | HotpotQA + SquadV2  | 0.16 | 0.08 | 0.15 | 0.35 | 0.20 |
 
 ## Contact
 
-[Maeda Hanafi](mailto:maeda.hanafi@ibm.com)
+[Lihong He](mailto:lihong.he@ibm.com)
 
 ### Framework versions