ibm-granite
/

granitelib-rag-r1.0

@@ -35,238 +35,36 @@ This is a family of hallucination detection intrinsics that gives the ability to
 ## Quickstart Example
-To run the hallucination detection intrinsics through granite-common, you can either (a) use an OpenAI-compatible inference backend, such as vLLM or (b) use the Hugging Face Transformers library. We provide below instructions for each of the two approaches. Note that running inference using vLLM or another scalable OpenAI-compatible inference backend should be significantly faster than using the Hugging Face Transformers library directly.
-### Using an OpenAI-Compatible Inference Backend
-To run the intrinsic using an OpenAI-compatible inference backend, such as vLLM, follow the steps below. We recommend using Python 3.11 or higher.
-1. Install the granite-common library and nltk:
-   ```
-   pip install git+https://github.com/ibm-granite/granite-common.git
-   pip install nltk
-   ```
-2. Install the Hugging Face CLI:
-   ```
-   pip install -U "huggingface_hub[cli]"
-   ```
-3. Install vLLM:
-   ```
-   pip install vllm
-   ```
-4. Download the intrinsics library:
-   ```
-   hf download ibm-granite/rag-intrinsics-lib --local-dir ./rag-intrinsics-lib
-   ```
-5. Edit the vLLM startup script found in `./rag-intrisics-lib/run_vllm.sh` using your favorite editor:
-   Edit the constants `BASE_MODEL_NAME` and `BASE_MODEL_ORG` depending on the base model on which the desired LoRA adapter has been trained. Optionally, edit the constant `PORT` to change the port on which vLLM will run. Save the modified file and exit the editor.
-6. Start vLLM through the startup script. The first time you run the script, you may have to change the permissions to allow execution:
-   ```
-   cd rag-intrinsics-lib
-   chmod u+x ./run_vllm.sh
-   ./run_vllm.sh &
-   ```
-7. Run the following code snippet:
-   ```
-    import json
-    import openai
-    import granite_common
-    intrinsic_name = "hallucination_detection"
-    # Change the following constant to select a different base model
-    base_model_name = "granite-3.3-8b-instruct"
-    # Change the following constants as needed to reflect the location of the vLLM server
-    # The selected port should be identical to the one you specified in the vLLM startup script
-    openai_base_url = "http://p2-r15-n1.bluevela.rmf.ibm.com:55555/v1"
-    openai_api_key = "rag_intrinsics_1234"
-    # Fetch IO configuration file from Hugging Face Hub
-    io_yaml_file = granite_common.intrinsics.util.obtain_io_yaml(
-        intrinsic_name, base_model_name
     )
-    # Instantiate input/output processors
-    rewriter = granite_common.IntrinsicsRewriter(config_file=io_yaml_file)
-    result_processor = granite_common.IntrinsicsResultProcessor(config_file=io_yaml_file)
-    # Sample request
-    request_json = {
-        "messages": [
-            {
-                "role": "user",
-                "content": "What happened to Dennis Wilson of the Beach Boys in 1983?"
-            },
-            {
-                "role": "assistant",
-                "content": "On December 28, 1983, Dennis Wilson of the Beach Boys drowned in Marina del Rey while diving from a friend’s boat in an attempt to retrieve belongings he had earlier thrown. Forensic pathologists believed that Dennis experienced shallow-water blackout just before his death."
-            }
-        ],
-        "extra_body": {
-            "documents": [
-                {
-                    "doc_id": "0",
-                    "text": "The Beach Boys are an American rock band formed in Hawthorne, California, in 1961. The group's original lineup consisted of brothers Brian, Dennis, and Carl Wilson; their cousin Mike Love; and their friend Al Jardine. Distinguished by their vocal harmonies and early surf songs, they are one of the most influential acts of the rock era. The band drew on the music of jazz-based vocal groups, 1950s rock and roll, and black R&B to create their unique sound, and with Brian as composer, arranger, producer, and de facto leader, often incorporated classical or jazz elements and unconventional recording techniques in innovative ways. In 1983, tensions between Dennis and Love escalated so high that each obtained a restraining order against each other. With the rest of the band fearing that he would end up like Brian, Dennis was given an ultimatum after his last performance in November 1983 to check into rehab for his alcohol problems or be banned from performing live with them. Dennis checked into rehab for his chance to get sober, but on December 28, 1983, he fatally drowned in Marina del Rey while diving from a friend's boat trying to recover items that he had previously thrown overboard in fits of rage."
-                },
-                {
-                    "doc_id": "1",
-                    "text": "A cigarette smoker since the age of 13, Carl was diagnosed with lung cancer after becoming ill at his vacation home in Hawaii, in early 1997. Despite his illness, Carl continued to perform while undergoing chemotherapy. He played and sang throughout the Beach Boys' entire summer tour which ended in the fall of 1997. During the performances, he sat on a stool, but he stood while singing \"God Only Knows\".  Carl died of lung cancer in Los Angeles, surrounded by his family, on February 6, 1998, just two months after the death of his mother, Audree Wilson. He was interred at Westwood Village Memorial Park Cemetery in Los Angeles."
-                },
-                {
-                    "doc_id": "2",
-                    "text": "Carl Dean Wilson (December 21, 1946 - February 6, 1998) was an American musician, singer, and songwriter who co-founded the Beach Boys. He is best remembered as their lead guitarist, as the youngest brother of bandmates Brian and Dennis Wilson, and as the group's de facto leader in the early 1970s. He was also the band's musical director on stage from 1965 until his death. Influenced by the guitar playing of Chuck Berry and the Ventures, Carl's initial role in the group was that of lead guitarist and backing vocals, but he performed lead vocals on several of their later hits, including \"God Only Knows\" (1966), \"Good Vibrations\" (1966), and \"Kokomo\" (1988). By the early 1980s the Beach Boys were in disarray; the band had split into several camps. Frustrated with the band's sluggishness to record new material and reluctance to rehearse, Wilson took a leave of absence in 1981.  He quickly recorded and released a solo album, Carl Wilson, composed largely of rock n' roll songs co-written with Myrna Smith-Schilling, a former backing vocalist for Elvis Presley and Aretha Franklin, and wife of Wilson's then-manager Jerry Schilling. The album briefly charted, and its second single, \"Heaven\", reached the top 20 on Billboard's Adult Contemporary chart."
-                }
-            ]
-        }
-    }
-    # Add other parameters
-    request_json["model"] = intrinsic_name
-    request_json["temperature"] = 0.0
-    # Apply input processor
-    rewritten_request = rewriter.transform(request_json)
-    # Run inference
-    client = openai.OpenAI(base_url=openai_base_url, api_key=openai_api_key)
-    chat_completion = client.chat.completions.create(**rewritten_request.model_dump())
-    # Apply output processor
-    processed_chat_completion = result_processor.transform(
-        chat_completion, rewritten_request
-    )
-    # Verify that the contents of the completion is valid JSON and pretty-print the JSON.
-    parsed_contents = json.loads(processed_chat_completion.choices[0].message.content)
-    print("JSON output:")
-    print(json.dumps(parsed_contents, indent=2))
-   ```
-### Using the Hugging Face Transformers Library
-To run the intrinsic using the Hugging Face Transformers library directly, follow the steps below. We recommend using Python 3.11 or higher.
-1. Install the granite-common library and nltk:
-   ```
-   pip install git+https://github.com/ibm-granite/granite-common.git
-   pip install nltk
-   ```
-2. Install the Hugging Face CLI:
-   ```
-   pip install -U "huggingface_hub[cli]"
-   ```
-3. Install PEFT:
-   ```
-   pip install peft
-   ```
-4. Install xgrammar:
-   ```
-   pip install xgrammar
-   ```
-5. Run the following code snippet:
-   ```
-   import json
-   import granite_common.util
-   import peft
-   intrinsic_name = "hallucination_detection"
-   # Change the following constant to select a different base model
-   base_model_name = "granite-3.3-8b-instruct"
-   use_cuda = True  # Set to False to use default PyTorch device for this machine + model
-   # Fetch IO configuration file from Hugging Face Hub
-   io_yaml_file = granite_common.intrinsics.util.obtain_io_yaml(
-       intrinsic_name, base_model_name
-   )
-   # Fetch LoRA directory from Hugging Face Hub
-   lora_dir = granite_common.intrinsics.util.obtain_lora(
-       intrinsic_name, base_model_name
-   )
-   # Instantiate input/output processors
-   rewriter = granite_common.IntrinsicsRewriter(config_file=io_yaml_file)
-   result_processor = granite_common.IntrinsicsResultProcessor(config_file=io_yaml_file)
-   # Sample request
-   request_json = {
-       "messages": [
-           {
-               "role": "user",
-               "content": "What is the visibility level of Git Repos and Issue Tracking projects?"
-           },
-           {
-               "role": "assistant",
-               "content": "Git Repos and Issue Tracking projects can have one of the following visibility levels: private, internal, or public. Private projects are visible only to project members, internal projects are visible to all users that are logged in to IBM Cloud, and public projects are visible to anyone. By default, new projects are set to private visibility level, which is the most secure for your data."
-           }
-       ],
-       "extra_body": {
-           "documents": [
-               {
-                   "doc_id": "0",
-                   "text": "Git Repos and Issue Tracking is an IBM-hosted component of the Continuous Delivery service. All of the data that you provide to Git Repos and Issue Tracking, including but not limited to source files, issues, pull requests, and project configuration properties, is managed securely within Continuous Delivery. However, Git Repos and Issue Tracking supports various mechanisms for exporting, sending, or otherwise sharing data to users and third parties. The ability of Git Repos and Issue Tracking to share information is typical of many social coding platforms. However, such sharing might conflict with regulatory controls that apply to your business. After you create a project in Git Repos and Issue Tracking, but before you entrust any files, issues, records, or other data with the project, review the project settings and change any settings that you deem necessary to protect your data. Settings to review include visibility levels, email notifications, integrations, web hooks, access tokens, deploy tokens, and deploy keys. Project visibility levels \n\nGit Repos and Issue Tracking projects can have one of the following visibility levels: private, internal, or public. * Private projects are visible only to project members. This setting is the default visibility level for new projects, and is the most secure visibility level for your data. * Internal projects are visible to all users that are logged in to IBM Cloud. * Public projects are visible to anyone. To limit project access to only project members, complete the following steps:\n\n\n\n1. From the project sidebar, click Settings > General. 2. On the General Settings page, click Visibility > project features > permissions. 3. Locate the Project visibility setting. 4. Select Private, if it is not already selected. 5. Click Save changes. Project membership \n\nGit Repos and Issue Tracking is a cloud hosted social coding environment that is available to all Continuous Delivery users. If you are a Git Repos and Issue Tracking project Maintainer or Owner, you can invite any user and group members to the project. IBM Cloud places no restrictions on who you can invite to a project."
-               },
-               {
-                   "doc_id": "1",
-                   "text": "After you create a project in Git Repos and Issue Tracking, but before you entrust any files, issues, records, or other data with the project, review the project settings and change any settings that are necessary to protect your data. Settings to review include visibility levels, email notifications, integrations, web hooks, access tokens, deploy tokens, and deploy keys. Project visibility levels \n\nGit Repos and Issue Tracking projects can have one of the following visibility levels: private, internal, or public. * Private projects are visible only to project members. This setting is the default visibility level for new projects, and is the most secure visibility level for your data. * Internal projects are visible to all users that are logged in to IBM Cloud. * Public projects are visible to anyone. To limit project access to only project members, complete the following steps:\n\n\n\n1. From the project sidebar, click Settings > General. 2. On the General Settings page, click Visibility > project features > permissions. 3. Locate the Project visibility setting. 4. Select Private, if it is not already selected. 5. Click Save changes. Project email settings \n\nBy default, Git Repos and Issue Tracking notifies project members by way of email about project activities. These emails typically include customer-owned data that was provided to Git Repos and Issue Tracking by users. For example, if a user posts a comment to an issue, Git Repos and Issue Tracking sends an email to all subscribers. The email includes information such as a copy of the comment, the user who posted it, and when the comment was posted. To turn off all email notifications for your project, complete the following steps:\n\n\n\n1. From the project sidebar, click Settings > General. 2. On the **General Settings **page, click Visibility > project features > permissions. 3. Select the Disable email notifications checkbox. 4. Click Save changes. Project integrations and webhooks"
-               }
-           ]
-       }
-   }
-   # Add additional parameters
-   request_json["model"] = intrinsic_name
-   request_json["temperature"] = 0.0
-   # Apply input processor
-   rewritten_request = rewriter.transform(request_json)
-   # Load the base model and merge LoRA weights
-   model, tokenizer = granite_common.util.load_transformers_lora(lora_dir)
-   if use_cuda:
-       model = model.cuda()
-   # Convert the chat completion request into a the Transformers library's proprietary
-   # format.
-   generate_input, other_input = (
-       granite_common.util.chat_completion_request_to_transformers_inputs(
-           rewritten_request,
-           tokenizer,
-           model,
-       )
-   )
-   # Use the Transformers library's APIs to generate one or more completions,
-   # then convert those completions into OpenAI-compatible chat completion
-   responses = granite_common.util.generate_with_transformers(
-       tokenizer, model, generate_input, other_input
-   )
-   # Apply output processor
-   transformed_responses = result_processor.transform(responses, rewritten_request)
-   # Verify that the contents of the completion is valid JSON and pretty-print the JSON.
-   parsed_contents = json.loads(transformed_responses.choices[0].message.content)
-   print("JSON output:")
-   print(json.dumps(parsed_contents, indent=2))
-   ```
 ## Training Details

 ## Quickstart Example
+The recommended way to call this intrinsic is through the [Mellea](https://mellea.ai) framework.
+Here is some example code for calling this intrinsic from Mellea:
+```
+from mellea.backends.huggingface import LocalHFBackend
+from mellea.stdlib.base import ChatContext, Document
+from mellea.stdlib.chat import Message
+from mellea.stdlib.intrinsics import rag
+import json
+backend = LocalHFBackend(model_id="ibm-granite/granite-3.3-2b-instruct")
+context = (
+    ChatContext()
+    .add(Message("assistant", "Hello there, how can I help you?"))
+    .add(Message("user", "Tell me about some yellow fish."))
+)
+assistant_response = "Purple bumble fish are yellow. Green bumble fish are also yellow."
+documents = [
+    Document(
+        doc_id="1",
+        text="The only type of fish that is yellow is the purple bumble fish.",
     )
+]
+result = rag.flag_hallucinated_content(assistant_response, documents, context, backend)
+print(f"Result of hallucination check: {json.dumps(result, indent=2)}")
+```
 ## Training Details