Update citation generation model card with latest results and reference to granite-4.0-micro model card (#7)

- Update citation generation model card with latest results and reference to granite-4.0-micro model card (1fd194df60b990691b66a30fd3b21bf5a1d2175e)

Co-authored-by: Yannis Katsis <yannisk2@users.noreply.huggingface.co>

Files changed (1) hide show

citations/README.md +154 -147

citations/README.md CHANGED Viewed

@@ -7,208 +7,215 @@ library_name: peft
 library_name: transformers
 ---
-# Intrinsics for Citation Generation
 ## Model Summary
-This is a RAG-specific family of intrinsics fine-tuned for the citation generation task. Given a multi-turn conversation between a user and an AI assistant ending with an assistant response and a set of documents/passages on which the last assistant response is supposed to be based, each intrinsic in the family generates citations for the last assistant response from the provided documents/passages. The intrinsic has the following features:
-1. **Fine-grained citations:** The intrinsic generates citations for each sentence in the assistant response (when available). Moreover, each citation consists of a set of sentences from the documents/passages that support the corresponding sentence in the assistant response.
-2. **Post-hoc citation generation:** Since the intrinsic takes the assistant response as input, it can generate citations for responses generated by any LLM. Pick your favorite LLM and use the intrinsic to generate post-hoc citations!
-We provide two intrinsics implemented as LoRA adapters trained over Granite-3.3-2b-instruct and Granite-3.3-8b-instruct, respectively.
-</br>
 - **Developer:** IBM Research
-- **Model type:** LoRA adapter for [ibm-granite/granite-3.3-2b-instruct](https://huggingface.co/ibm-granite/granite-3.3-2b-instruct) and [ibm-granite/granite-3.3-8b-instruct](https://huggingface.co/ibm-granite/granite-3.3-8b-instruct)
 - **License:** [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0)
 ## Intended use
-This is a family of citation generation intrinsics that give the ability to generate citations for the last assistant response in a multi-turn RAG conversation based on a set of provided documents/passages. They can be used to generate post-hoc citations for assistant responses generated by any LLM in a RAG setting.
 > [!TIP]
-> Note: While you can invoke a citation generation intrinsic directly, it is strongly recommended to call it through [granite-common](https://github.com/ibm-granite/granite-common), which wraps the model with a tailored I/O processor, enabling a friendlier development interface. The I/O processor takes care of several data transformation/validation tasks that would be otherwise required (incl. splitting the input documents and assistant response into sentences before calling the intrinsic as well as validating the intrinsic's output and transforming the returned sentence IDs into spans over the documents and the response). We next describe the input/output of the citation generation intrinsics when invoked through granite-common.
-**Intrinsic input**: The input to the citation generation intrinsic is an OpenAI-compatible chat completion request, containing a list of conversation turns ending with the assistant response for which the citations should be generated as well as the list of documents from which the citations should be drawn. Please see the code snippets in the Quickstart Example section below for examples on how to specify the chat completion request as a JSON object.
-**Intrinsic output**: The output of the citation generation intrinsic is formatted as the result of the original chat completion request containing the citations for the last assistant response. The citations are provided in the form of a JSON array, whose items include the text and begin/end of a response span together with the text, document id and begin/end of a document span that serves as a citation for the response span. When there are more than one document spans that serve as citations for a single response span, they are represented as separate objects in the JSON array.
-**Going from input to output**: When calling the intrinsic through granite-common one should follow the steps below to transform the intrinsic input to the corresponding output. These steps are also exemplified in the code snippets included in the Quickstart Example section below. Given an input chat completion request, the request should be  passed to the corresponding input processor (also referred to as IntrinsicsRewriter) provided by granite-common. The input processor converts the request to the appropriate format expected by the underlying citation generation model. This includes, among others, splitting the last assistant response and the documents into sentences and prepending them with sentence IDs as well as introducing an appropriate task-specific instruction. The input processor's result should then be passed to the underlying citation generation model for inference. The model generates citations using a compact representation consisting of sentence IDs in the last assistant response and documents. This output should finally be passed to the appropriate output processor (also referred to as IntrinsicsResultProcessor) provided by granite-common. The output processor converts the low-level raw model output to the final output by, among others, mapping the sentence IDs back to response and document spans. The result is an application-friendly format ready for consumption by downstream applications.
-## Quickstart Example
-The recommended way to call this intrinsic is through the [Mellea](https://mellea.ai) framework.
-Here is some example code for calling this intrinsic from Mellea:
-```
-from mellea.backends.huggingface import LocalHFBackend
-from mellea.stdlib.base import ChatContext, Document
-from mellea.stdlib.chat import Message
-from mellea.stdlib.intrinsics import rag
-import json
-backend = LocalHFBackend(model_id="ibm-granite/granite-3.3-2b-instruct")
-context = ChatContext().add(
-    Message(
-        "user",
-        "How does Murdoch's expansion in Australia compare to his expansion "
-        "in New Zealand?",
-    )
-)
-assistant_response = (
-    "Murdoch expanded in Australia and New Zealand by acquiring and expanding local "
-    "newspapers. I do not have information about his expansion in New Zealand after "
-    "purchasing The Dominion."
-)
-documents = [
-    Document(
-        doc_id="1",
-        text="Keith Rupert Murdoch was born on 11 March 1931 in Melbourne, Australia, "
-        "the son of Sir Keith Murdoch (1885-1952) and Dame Elisabeth Murdoch (nee "
-        "Greene; 1909-2012). He is of English, Irish, and Scottish ancestry. Murdoch's "
-        "parents were also born in Melbourne. Keith Murdoch was a war correspondent "
-        "and later a regional newspaper magnate owning two newspapers in Adelaide, "
-        "South Australia, and a radio station in a faraway mining town. Following his "
-        "father's death, when he was 21, Murdoch returned from Oxford to take charge "
-        "of the family business News Limited, which had been established in 1923. "
-        "Rupert Murdoch turned its Adelaide newspaper, The News, its main asset, into "
-        "a major success. He began to direct his attention to acquisition and "
-        "expansion, buying the troubled Sunday Times in Perth, Western Australia "
-        "(1956) and over the next few years acquiring suburban and provincial "
-        "newspapers in New South Wales, Queensland, Victoria and the Northern "
-        "Territory, including the Sydney afternoon tabloid, The Daily Mirror (1960). "
-        'The Economist describes Murdoch as "inventing the modern tabloid", as he '
-        "developed a pattern for his newspapers, increasing sports and scandal "
-        "coverage and adopting eye-catching headlines. Murdoch's first foray outside "
-        "Australia involved the purchase of a controlling interest in the New Zealand "
-        "daily The Dominion. In January 1964, while touring New Zealand with friends "
-        "in a rented Morris Minor after sailing across the Tasman, Murdoch read of a "
-        "takeover bid for the Wellington paper by the British-based Canadian newspaper "
-        "magnate, Lord Thomson of Fleet. On the spur of the moment, he launched a "
-        "counter-bid. A four-way battle for control ensued in which the 32-year-old "
-        "Murdoch was ultimately successful. Later in 1964, Murdoch launched The "
-        "Australian, Australia's first national daily newspaper, which was based "
-        "first in Canberra and later in Sydney. In 1972, Murdoch acquired the Sydney "
-        "morning tabloid The Daily Telegraph from Australian media mogul Sir Frank "
-        "Packer, who later regretted selling it to him. In 1984, Murdoch was appointed "
-        "Companion of the Order of Australia (AC) for services to publishing. In 1999, "
-        "Murdoch significantly expanded his music holdings in Australia by acquiring "
-        "the controlling share in a leading Australian independent label, Michael "
-        "Gudinski's Mushroom Records; he merged that with Festival Records, and the "
-        "result was Festival Mushroom Records (FMR). Both Festival and FMR were "
-        "managed by Murdoch's son James Murdoch for several years.",
-    ),
-    Document(
-        doc_id="2",
-        text="This document has nothing to do with Rupert Murdoch. This document is "
-        "two sentences long.",
-    ),
 ]
-result = rag.find_citations(assistant_response, documents, context, backend)
-print(f"Result of citations intrinsic:\n{json.dumps(result, indent=2)}")
-```
 ## Training Details
-The citation generation intrinsics were trained on synthetically-generated citation datasets. The process of generating the training data consisted of two main steps:
-- **Multi-turn RAG conversation generation:** Starting from publicly available document corpora, we generated a set of multi-turn RAG data, consisting of multi-turn conversations grounded on passages retrieved from the corpora. For details on the RAG conversation generation process please refer to the [Granite Technical Report](https://github.com/ibm-granite/granite-3.0-language-models/blob/main/paper.pdf) and [Lee, Young-Suk, et al.](https://arxiv.org/pdf/2409.11500).
-- **Citation generation:** For each turn of the multi-turn RAG conversations from the previous step, we used a multi-step synthetic citation generation pipeline to generate citations for the assistant response.
-The resulting data instances were used to train the citation generation intrinsics.
 ### Training Data
 The following public datasets were used as seed datasets for the multi-turn RAG conversation generation process:
-- [CoQA](https://stanfordnlp.github.io/coqa/) - Wikipedia passages
 - [MultiDoc2Dial](https://huggingface.co/datasets/IBM/multidoc2dial)
 - [QuAC](https://huggingface.co/datasets/allenai/quac)
 ## Evaluation
-We evaluate the citation generation intrinsics on two citation benchmarks:
-- [ALCE](https://aclanthology.org/2023.emnlp-main.398/): Evaluates the ability of models to produce document/passage-level citations (i.e., identify the documents/passages that support a statement in the response).
-- [LongBench-Cite](https://arxiv.org/abs/2409.02897): Evaluates the ability of models to produce fine-grained span-level citations (i.e., identify the spans within the input documents/passages that support a statement in the response) with a focus on long contexts.
-Since the intrinsics correspond to a post-hoc citation generation approach, their performance on the two benchmarks depends on the assistant responses for which they are asked to generate citations. To facilitate an apples-to-apples comparison, for each experiment, we keep the assistant responses the same and change the model that is used to generate the citations. In particular, we prompt an LLM to create an assistant response together with citations and evaluate the generated citations on the corresponding benchmark. Then, we compute and evaluate the citations generated for the same LLM response by each of the citation generation intrinsics. We provide results for the two intrinsics, implemented as LoRA adapters over Granite-3.3-2b-instruct and Granite-3.3-8b-instruct, respectively.
-### Evaluation on ALCE
-For the ALCE evaluation, we prompt Llama-3.1-70B-Instruct and Mixtral-8x22B-Instruct to generate both the assistant response and corresponding passage-level citations. We first calculate the performance of the citations generated by these models on ALCE. Subsequently, we feed the responses of these models (leaving out the citations) to the citation generation intrinsics and evaluate their generated citations. The results are shown in the table below:
-Model used to generate response | Model used to generate citations                      | Recall          | Precision         |  F1       |
-|--------------| ----------------------------- | --------------- | ----------------- | --------- |
-| Llama-3.1-70B-Instruct | Llama-3.1-70B-Instruct        | 61.4            | 58.1              | 59.7      |
-| Llama-3.1-70B-Instruct | Granite-3.3-2B LoRA citations | 51.5            | 64.2              | 57.2      |
-| Llama-3.1-70B-Instruct | Granite-3.3-8B LoRA citations | 55.4            | 64.2              | 59.5      |
-| Mixtral-8x22B-Instruct | Mixtral-8x22B-Instruct        | 62.2            | 62.5              | 62.3      |
-| Mixtral-8x22B-Instruct | Granite-3.3-2B LoRA citations | 51.4            | 67.3              | 58.3      |
-| Mixtral-8x22B-Instruct | Granite-3.3-8B LoRA citations | 55.8            | 68.5              | 61.5      |
-We observe that the LoRA adapter over Granite-3.3-8b-instruct performs on par with much bigger models when those are prompted to create passage-level citations (with the LoRA adapter over over Granite-3.3-2b-instruct being slightly worse). It is interesting to note that while the adapter's F1 performance is similar to the baselines, it exhibits a different precision-recall trade-off, trading lower recall for higher precision.
-Notes:
-- All results are reported on the ELI5 dataset using the ORACLE (5-psg) setting.
-- To prompt Llama and Mixtral, we employ a setting similar to the one proposed in the ALCE paper; in particular we use a two-shot prompt comprised of two of the ICL examples from ALCE as well as a slightly modified version of the instruction from the paper.
-- Sentence splitting of context/response is performed using NLTK.
-- Finally, since ALCE expects passage-level citations, we elevate the finer-grained citations produced by the LoRA adapter to the passage level before running the ALCE evaluation.
-### Evaluation on LongBench-Cite
-For the LonBench-Cite evaluation, we prompt Llama-3.1-70B-Instruct to generate both the assistant response and corresponding citations. Then we evaluate the citations generated by Llama as well as the post-hoc citations generated by the citation generation intrinsics when invoked on the Llama responses. The results are shown in the table below:
 <table>
 <tr>
-    <th>Model used to generate response</th>
-    <th>Model used to generate citations</th>
     <th colspan="3">Longbench-Chat (en)</th>
     <th colspan="3">MultifieldQA (en)</th>
     <th colspan="3">HotpotQA</th>
     <th colspan="3">GovReport</th>
 </tr>
 <tr>
-    <th></th>
     <th></th>
     <th>R</th><th>P</th><th>F1</th>
     <th>R</th><th>P</th><th>F1</th>
     <th>R</th><th>P</th><th>F1</th>
     <th>R</th><th>P</th><th>F1</th>
 </tr>
 <tr>
-    <td>Llama-3.1-70B-Instruct</td>
-    <td>Llama-3.1-70B-Instruct</td>
-    <td>27.0</td><td>34.4</td><td>26.1</td>
-    <td>46.1</td><td>63.3</td><td>49.7</td>
-    <td>34.0</td><td>39.4</td><td>30.2</td>
-    <td>55.0</td><td>77.5</td><td>62.0</td>
 </tr>
 <tr>
-    <td>Llama-3.1-70B-Instruct</td>
-    <td>Granite-3.3-2B LoRA citations</td>
-    <td>38.7</td><td>47.4</td><td>39.3</td>
-    <td>66.4</td><td>81.8</td><td>70.4</td>
-    <td>60.7</td><td>68.5</td><td>59.7</td>
-    <td>60.1</td><td>72.4</td><td>64.7</td>
 </tr>
 <tr>
-    <td>Llama-3.1-70B-Instruct</td>
-    <td>Granite-3.3-8B LoRA citations</td>
-    <td>54.5</td><td>59.9</td><td>55.6</td>
-    <td>73.0</td><td>82.9</td><td>75.7</td>
-    <td>68.5</td><td>73.8</td><td>66.4</td>
-    <td>73.5</td><td>84.6</td><td>78.2</td>
 </tr>
 </table>
-We observe that both variants of the LoRA adapter (even the one trained over Granite-3.3-2b-instruct) perform across the board significantly better than Llama-3.1-70B-Instruct when prompted to create span-level citations. This demonstrates the value of the adapter to create post-hoc citations even for assistant responses generated by much bigger LLMs.
 Notes:
 - The evaluation results are reported on the English subset of LongBench-Cite (i.e., restricted to instances whose `language` field equals to `en`).
-- To prompt Llama to generate a response with citations, we use the one-shot prompt described in the paper.
-- For the LoRA adapter, sentence splitting of the context is performed using NLTK. For the response, we reuse the splitting in Llama's output (since the LongBench-Cite prompt instructs the model to output a response split into sentences/statements).
 ## Model Card Authors

 library_name: transformers
 ---
+# Citation Generation Intrinsic
 ## Model Summary
+This is a RAG-specific intrinsic fine-tuned for the citation generation task. Given a multi-turn conversation between a user and an AI assistant ending with an assistant response and a set of documents/passages on which the last assistant response is supposed to be based, the intrinsic generates citations for the last assistant response from the provided documents/passages. The citation generation intrinsic has the following features:
+1. **Fine-grained citations:** The intrinsic generates citations for each sentence of the assistant response (when available). Each citation consists of a set of sentences from the documents/passages that support the corresponding sentence in the assistant response.
+2. **Post-hoc citation generation:** Since the intrinsic takes the assistant response as input, it can generate citations for responses generated by any LLM. Pick your favorite LLM for response generation and use the citation generation intrinsic to generate post-hoc citations!
+We have created two implementation of the intrinsic as LoRA adapters trained over granite-4.0-micro and gpt-oss-20b, respectively. This is the model card for the LoRA adapter trained over gpt-oss-20b. The model card for the LoRA adapter trained over granite-4.0-micro can be found [here](https://huggingface.co/ibm-granite/granitelib-rag-r1.0/blob/main/citations/README.md).
 - **Developer:** IBM Research
+- **Model type:** LoRA adapter for [openai/gpt-oss-20b](https://huggingface.co/openai/gpt-oss-20b)
 - **License:** [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0)
 ## Intended use
+This is a citation generation intrinsic that gives the ability to generate citations for the last assistant response in a multi-turn RAG conversation based on a set of provided documents/passages. It can be used to generate post-hoc citations for assistant responses generated by any LLM in a RAG setting.
 > [!TIP]
+> Note: While you can invoke the citation generation intrinsic directly, it is strongly recommended to call it through the [Mellea](https://mellea.ai) framework, which wraps the model with a tailored I/O processor, enabling a friendlier development interface. We next describe the input/output of the citation generation intrinsic when invoked through Mellea.
+**Intrinsic input**: The citation generation intrinsic takes as input the following:
+- _Conversation:_ A list of conversational turns ending with the last user question, encoded as a list of user/assistant messages.
+- _Assistant response:_ The assistant response to the last user question, which is also the response for which citations will be generated, provided as a string.
+- _Documents:_ A list of documents from which the citations should be drawn, encoded as a collection of Document objects.
+**Intrinsic output**: The output of the citation generation intrinsic contains the citations for the last assistant response. The citations are provided in the form of a JSON array, whose items include the text and begin/end of a response span together with the text, document id and begin/end of a document span that serves as a citation for that response span. When there are more than one document spans that serve as citations for a single response span, they are represented as separate objects in the JSON array.
+**Going from input to output**: When calling the intrinsic through Mellea, the framework internally performs multiple steps to transform the intrinsic input to the corresponding output. While you do not have to explicitly invoke these steps, we next provide a brief overview of this process. Given an input to the citation generation intrinsic, Mellea performs the following tasks:
+- _Convert user input to the appropriate format expected by the underlying citation generation model._ This includes, among others, splitting the last assistant response and the documents into sentences and prepending them with sentence IDs as well as introducing an appropriate task-specific instruction.
+- _Call underlying citation generation model for inference._ The model generates citations using a compact representation consisting of sentence IDs in the last assistant response and documents.
+- _Convert model output to final output._ The low-level raw model output is converted to the final output by, among others, mapping the sentence IDs back to response and document spans. The result is an application-friendly JSON format ready for consumption by downstream applications.
+## Example
+You can find below an example of the input and corresponding output of the citations generation intrinsic:
+### Input
+**Conversation:**
+User: What is the visibility level of Git Repos and Issue Tracking projects?
+**Assistant response:**
+Git Repos and Issue Tracking projects can have one of the following visibility levels: private, internal, or public. Private projects are visible only to project members, internal projects are visible to all users that are logged in to IBM Cloud, and public projects are visible to anyone. By default, new projects are set to private visibility level, which is the most secure for your data.
+**Documents:**
+Git Repos and Issue Tracking is an IBM-hosted component of the Continuous Delivery service. All of the data that you provide to Git Repos and Issue Tracking, including but not limited to source files, issues, pull requests, and project configuration properties, is managed securely within Continuous Delivery. However, Git Repos and Issue Tracking supports various mechanisms for exporting, sending, or otherwise sharing data to users and third parties. The ability of Git Repos and Issue Tracking to share information is typical of many social coding platforms. However, such sharing might conflict with regulatory controls that apply to your business. After you create a project in Git Repos and Issue Tracking, but before you entrust any files, issues, records, or other data with the project, review the project settings and change any settings that you deem necessary to protect your data. Settings to review include visibility levels, email notifications, integrations, web hooks, access tokens, deploy tokens, and deploy keys. Project visibility levels \n\nGit Repos and Issue Tracking projects can have one of the following visibility levels: private, internal, or public. * Private projects are visible only to project members. This setting is the default visibility level for new projects, and is the most secure visibility level for your data. * Internal projects are visible to all users that are logged in to IBM Cloud. * Public projects are visible to anyone. To limit project access to only project members, complete the following steps:\n\n\n\n1. From the project sidebar, click Settings > General. 2. On the General Settings page, click Visibility > project features > permissions. 3. Locate the Project visibility setting. 4. Select Private, if it is not already selected. 5. Click Save changes. Project membership \n\nGit Repos and Issue Tracking is a cloud hosted social coding environment that is available to all Continuous Delivery users. If you are a Git Repos and Issue Tracking project Maintainer or Owner, you can invite any user and group members to the project. IBM Cloud places no restrictions on who you can invite to a project.
+### Output
+```json
+[
+    {
+        "response_begin": 0,
+        "response_end": 117,
+        "response_text": "Git Repos and Issue Tracking projects can have one of the following visibility levels: private, internal, or public. ",
+        "citation_doc_id": "1",
+        "citation_begin": 1034,
+        "citation_end": 1179,
+        "citation_text": "Project visibility levels \n\nGit Repos and Issue Tracking projects can have one of the following visibility levels: private, internal, or public. "
+    },
+    {
+        "response_begin": 117,
+        "response_end": 290,
+        "response_text": "Private projects are visible only to project members, internal projects are visible to all users that are logged in to IBM Cloud, and public projects are visible to anyone. ",
+        "citation_doc_id": "1",
+        "citation_begin": 1179,
+        "citation_end": 1235,
+        "citation_text": "* Private projects are visible only to project members. "
+    },
+    {
+        "response_begin": 117,
+        "response_end": 290,
+        "response_text": "Private projects are visible only to project members, internal projects are visible to all users that are logged in to IBM Cloud, and public projects are visible to anyone. ",
+        "citation_doc_id": "1",
+        "citation_begin": 1353,
+        "citation_end": 1472,
+        "citation_text": "* Internal projects are visible to all users that are logged in to IBM Cloud. * Public projects are visible to anyone. "
+    },
+    {
+        "response_begin": 290,
+        "response_end": 391,
+        "response_text": "By default, new projects are set to private visibility level, which is the most secure for your data.",
+        "citation_doc_id": "1",
+        "citation_begin": 1235,
+        "citation_end": 1353,
+        "citation_text": "This setting is the default visibility level for new projects, and is the most secure visibility level for your data. "
+    }
 ]
+```
+## Quickstart
+The recommended way to call this intrinsic is through the [Mellea](https://mellea.ai) framework. For code snippets demonstrating how to use this and other intrinsics, please refer to the [Mellea intrinsics examples](https://github.com/generative-computing/mellea/tree/main/docs/examples/intrinsics).
 ## Training Details
+The citation generation intrinsic was trained on synthetically-generated citation datasets. The process of generating the training data consisted of two main steps:
+- _Multi-turn RAG conversation generation:_ Starting from publicly available document corpora, we generated a set of multi-turn RAG data, consisting of multi-turn conversations grounded on passages retrieved from the corpora. For details on the RAG conversation generation process please refer to the [Granite Technical Report](https://github.com/ibm-granite/granite-3.0-language-models/blob/main/paper.pdf) and [Lee, Young-Suk, et al.](https://arxiv.org/pdf/2409.11500)
+- _Citation generation:_ For each turn of the multi-turn RAG conversations from the previous step, we used a multi-step synthetic citation generation pipeline to generate citations for the assistant response.
+The resulting data instances were used to train the citation generation intrinsic.
 ### Training Data
 The following public datasets were used as seed datasets for the multi-turn RAG conversation generation process:
 - [MultiDoc2Dial](https://huggingface.co/datasets/IBM/multidoc2dial)
 - [QuAC](https://huggingface.co/datasets/allenai/quac)
 ## Evaluation
+We evaluated the citation generation intrinsic on a revised version of the [LongBench-Cite](https://arxiv.org/abs/2409.02897) benchmark; a benchmark evaluating the ability of models to produce fine-grained span-level citations (i.e., identify the spans within the input documents/passages that support a statement in the response) with a focus on long contexts. Being originally designed to evaluate inline citation generation approaches (i.e., approaches generating the assistant response and the citations at the same time), we adapt the benchmark for the evaluation of post-hoc citation generation (where citations are generated for a given assistant response generated by an upstream model).
+For the following experiments, we prompted Llama-3.1-70B-Instruct to generate the assistant response for the LongBench-Cite tasks. Then, two types of models were asked to create citations for these responses:
+- _Citation generation LoRA adapters:_ These are the two citation generation LoRA adapter implementations of the citation intrinsic, as described above.
+- _Prompt-based baselines:_ These are out-of-the-box LLMs prompted to generate post-hoc citations for the given assistant responses. Prompting was performed through a version of the 1-shot prompt used in the original benchmark, adapted for post-hoc citation generation.
+The evaluation results are shown in the table below:
 <table>
 <tr>
+    <th>Model</th>
     <th colspan="3">Longbench-Chat (en)</th>
     <th colspan="3">MultifieldQA (en)</th>
     <th colspan="3">HotpotQA</th>
     <th colspan="3">GovReport</th>
+    <th>AVG F1</th>
 </tr>
 <tr>
     <th></th>
     <th>R</th><th>P</th><th>F1</th>
     <th>R</th><th>P</th><th>F1</th>
     <th>R</th><th>P</th><th>F1</th>
     <th>R</th><th>P</th><th>F1</th>
+    <th></th>
 </tr>
 <tr>
+    <th colspan="14" style="background-color: #f5f5f5;">Citation Generation LoRA Adapters</th>
 </tr>
 <tr>
+    <td>granite-4.0-micro LoRA</td>
+    <td>42.7</td><td>46.5</td><td>41.4</td>
+    <td>68.5</td><td>81.1</td><td>72.0</td>
+    <td>62.9</td><td>67.9</td><td>61.0</td>
+    <td>70.2</td><td>79.3</td><td>74.1</td>
+    <td><b>62.1</b></td>
 </tr>
 <tr>
+    <td>gpt-oss-20b LoRA</td>
+    <td>56.1</td><td>61.4</td><td>55.3</td>
+    <td>71.6</td><td>87.1</td><td>76.8</td>
+    <td>69.9</td><td>71.5</td><td>66.1</td>
+    <td>73.8</td><td>84.8</td><td>78.2</td>
+    <td><b>69.1</b></td>
 </tr>
+<tr>
+    <th></th>
+    <th></th><th></th><th></th>
+    <th></th><th></th><th></th>
+    <th></th><th></th><th></th>
+    <th></th><th></th><th></th>
+    <th></th>
+</tr>
+<tr>
+    <th colspan="14" style="background-color: #f5f5f5;">Prompting-based Baselines</th>
+</tr>
+<tr>
+    <td>granite-4.0-micro Prompted</td>
+    <td>5.4</td><td>6.1</td><td>3.5</td>
+    <td>22.3</td><td>32.1</td><td>22.3</td>
+    <td>18.0</td><td>21.4</td><td>14.2</td>
+    <td>8.9</td><td>18.0</td><td>10.1</td>
+    <td><b>12.5</b></td>
+</tr>
+<tr>
+    <td>gpt-oss-20b Prompted</td>
+    <td>38.0</td><td>38.7</td><td>34.7</td>
+    <td>56.8</td><td>68.1</td><td>59.5</td>
+    <td>54.0</td><td>60.4</td><td>52.2</td>
+    <td>48.3</td><td>59.2</td><td>52.4</td>
+    <td><b>49.7</b></td>
+</tr>
+<tr>
+    <td>gpt-oss-120b Prompted</td>
+    <td>46.4</td><td>49.1</td><td>45.2</td>
+    <td>68.9</td><td>76.6</td><td>70.1</td>
+    <td>65.4</td><td>66.5</td><td>62.2</td>
+    <td>70.8</td><td>75.4</td><td>72.2</td>
+    <td><b>62.4</b></td>
+</tr>
+<tr>
+    <td>gpt-4o Prompted</td>
+    <td>56.9</td><td>60.1</td><td>56.2</td>
+    <td>68.7</td><td>79.6</td><td>71.4</td>
+    <td>65.3</td><td>73.6</td><td>65.3</td>
+    <td>74.3</td><td>81.5</td><td>76.9</td>
+    <td><b>67.5</b></td>
+</tr>
 </table>
+We observe that both citation generation LoRA adapters perform better not only than the corresponding base models prompted out of the box but also better than bigger models. For instance, the granite-4.0-micro LoRA performs on par with prompting the significantly larger gpt-oss-120b. Similarly, the gpt-oss-20b LoRA outperforms prompting the much larger gpt-4o.
 Notes:
 - The evaluation results are reported on the English subset of LongBench-Cite (i.e., restricted to instances whose `language` field equals to `en`).
+- To generate the assistant responses fed to all evaluated models, we prompted Llama-3.1-70B-Instruct using the one-shot prompt described in the LongBench-Cite paper (which asks the model to generate a grounded response with citations) and removed the citations in post-processing.
+- The AVG F1 column contains the average of the four dataset-specific F1 scores.
 ## Model Card Authors