sammmmmmm25/alfred-ai-investigations

Alfred AI Investigations

1. Introduction

Criminal investigations generate evidence across many disconnected document types -- police reports, witness interview transcripts, call logs, evidence logs -- and the connections that matter most are often the ones that span across these documents rather than sitting inside any single one. An investigator re-reading a call log might miss that a contact appearing there also appears in an unrelated case's call log three weeks later, or that two witnesses in different burglaries independently described the same vehicle. General-purpose LLMs asked to reason over this kind of material either hallucinate connections that aren't there or miss real ones buried in documents the model was never shown.

Alfred AI Investigations is a Graph RAG assistant built to address this directly. Rather than treating case documents as an undifferentiated block of text for similarity search, Alfred extracts named entities from every document chunk (people, vehicles, locations, aliases, case numbers) and builds a graph connecting chunks that share an entity. At query time, Alfred combines standard embedding-based retrieval with one-hop graph traversal, surfacing cross-document connections that a flat vector search would miss because the connecting text is often lexically dissimilar (e.g., a phone contact saved as "R" in one document and a suspect's full alias in another). Every answer is required to cite its exact source document and section, and to explicitly flag when the retrieved evidence is insufficient or contradictory, rather than filling gaps with plausible-sounding fabrication.

This project deliberately avoided fine-tuning the generation model. Zero/few-shot evaluation in early development (see Methodology) showed that a sufficiently capable off-the-shelf instruction model already avoided fabricating citations and correctly surfaced timeline conflicts when given the right retrieved evidence -- the harder problem was getting the right evidence in front of the model, which is a retrieval problem, not a generation problem. That reframing is why this project's engineering effort went into the graph-retrieval architecture rather than model fine-tuning.

Across three evaluation sets, retrieval improved citation accuracy on Alfred's own held-out split from 0.00 (no retrieval) to 0.76, and improved answer correctness by 73% on LegalBench-RAG and 53% on RAGBench relative to no-RAG baselines. Compared against two similarly-sized instruction-tuned models run through the identical retrieval pipeline (Qwen2.5-3B-Instruct and Llama-3.2-3B-Instruct), the shipped Qwen3-4B-Instruct-2507 configuration won decisively on citation accuracy -- the metric most tied to Alfred's core goal of avoiding fabricated evidence -- even though the smaller comparison models occasionally scored higher on raw answer-text overlap.

2. Data

Alfred's corpus is a hand-authored synthetic set of four case files, each built from the same document structure real investigations produce: police reports, witness interview transcripts, call logs, and evidence logs, chunked by document section. The corpus deliberately encodes two cross-case connections that a competent investigator should be able to surface but that plain text search would likely miss:

• A phone contact ("R" / an area code) appears in the call logs of two separate cases (a burglary and an unrelated warehouse break-in), suggesting a possible common suspect.
• A witness description of a distinctive vehicle (a white pickup truck with a dented tailgate) recurs across two separate burglary cases, worded differently by each witness.

This corpus is intentionally small (4 cases, 20 chunks) rather than large and unverified -- see Limitations for the tradeoff this represents. An 8-query held-out test split, covering both single-hop (answerable from one document) and multi-hop (requiring the cross-case connections above) questions, was used for retrieval and generation evaluation throughout development.

To evaluate generalization beyond this narrow corpus, two external benchmarks were used:

• LegalBench-RAG (Pipitone & Houir Alami, 2024) -- the privacy_qa subset of LegalBench-RAG-mini (194 query-answer pairs), chosen for its citation-grounded, document-specific question format.
• RAGBench (Friel et al., 2025) -- the cuad (contract review) subset (510 test examples), chosen for its similarity to Alfred's document-grounded, citation-heavy task, and because it ships expert-annotated reference scores (adherence, relevance, completeness) alongside each example.

Neither external benchmark shares any topical overlap with Alfred's own corpus -- they serve purely as a check on whether the retrieval architecture generalizes, not as in-domain training or tuning data.

3. Methodology

Alfred's pipeline has two retrieval components that combine at query time:

1. Embedding + cosine similarity retrieval. Corpus chunks are embedded once at index time; at query time, the top-k most similar chunks are retrieved by cosine similarity.
2. Entity-graph expansion. During indexing, a fine-tuned NER model extracts entities from every chunk (8 types: person, organization, location, date, vehicle, weapon, case number, alias). Chunks sharing an entity -- by exact match or by fuzzy word-overlap match for same-type multi-word entities -- are linked in a graph. At query time, after the initial top-k retrieval, Alfred expands one hop through this graph, pulling in any chunk connected to something already retrieved.

Three retrieval configurations were compared on the held-out test split:

Combination	Embedding model	Graph expansion
Vector-only baseline	sentence-transformers/all-MiniLM-L6-v2	No
MiniLM + graph	sentence-transformers/all-MiniLM-L6-v2	Yes
BGE + graph (shipped)	BAAI/bge-small-en-v1.5	Yes

Generation model: Qwen/Qwen3-4B-Instruct-2507, selected after comparing it against smaller/similarly-sized alternatives in zero-, three-, and eight-shot settings. Qwen3-4B was the only model tested that never fabricated a citation and correctly surfaced a timeline conflict between witness testimony and call-log evidence in zero/few-shot evaluation, which made fine-tuning smaller models to reach that same baseline behavior an unattractive tradeoff given the project's hallucination-tolerance requirements. Alfred runs the generator at three-shot with a fixed system prompt (see Prompt Format).

NER model: dslim/bert-base-NER, fine-tuned on a small hand-authored BIO-tagged dataset covering the 8 entity types above. This model is the component most directly responsible for whether the entity graph actually connects the right chunks -- see Limitations for how its size affects both graph quality and run-to-run reproducibility. The fine-tuned model is pushed as its own repository, sammmmmmm25/alfred-ner-bert, and used as a subcomponent loaded by the main Alfred pipeline rather than bundled into this repo directly.

4. Evaluation

Alfred's own held-out test split, LegalBench-RAG, and RAGBench were each evaluated using a custom scoring suite built around the project's hallucination-first priorities rather than generic text-similarity metrics alone:

• Citation accuracy -- the fraction of cited documents that were actually part of the retrieved evidence (a fabricated citation is Alfred's worst failure mode).
• Format compliance -- whether the response follows the required Answer / Key Evidence / Cited Documents / Possible Leads / Limitations structure.
• Answer correctness -- ROUGE-L F1 against a gold reference response.
• Context recall -- the fraction of gold-relevant evidence chunks that were actually retrieved.

These three benchmarks were chosen because each targets a different weakness relevant to Alfred's task: Alfred's own split directly tests the multi-hop, cross-document reasoning the project was built around; LegalBench-RAG tests citation-grounded, document-specific question answering in a legal domain structurally similar to case files; and RAGBench (cuad subset) tests contract-style document grounding with independently-annotated reference scores, guarding against Alfred's own scoring suite being the only measure of success.

To satisfy the requirement of comparing against the base model and similarly-sized alternatives, two other instruction-tuned models of comparable size to Qwen/Qwen3-4B-Instruct-2507 were run through the identical retrieval pipeline (BGE-small embeddings + entity-graph expansion, same corpus, same few-shot prompt) with only the generator swapped out: Qwen2.5-3B-Instruct and Llama-3.2-3B-Instruct. Holding retrieval fixed isolates the generator's contribution -- this is why context recall is identical across all three generators within each benchmark below (recall depends only on what was retrieved, not on which model wrote the answer).

Alfred's own held-out test split

Model	Format compliance	Citation accuracy	Answer correctness	Context recall
Qwen3-4B-Instruct-2507 + RAG (shipped)	0.975	0.760	0.340	0.792
Qwen3-4B-Instruct-2507, no RAG (base model)	1.000	0.000	0.210	n/a
Qwen2.5-3B-Instruct + RAG	0.950	0.722	0.370	0.792
Llama-3.2-3B-Instruct + RAG	1.000	0.625	0.366	0.792

LegalBench-RAG (privacy_qa)

Model	Answer correctness	Context recall
Qwen3-4B-Instruct-2507 + RAG (shipped)	0.174	0.912
Qwen3-4B-Instruct-2507, no RAG (base model)	0.101	n/a
Qwen2.5-3B-Instruct + RAG	0.150	0.912
Llama-3.2-3B-Instruct + RAG	0.153	0.912

RAGBench (cuad)

Model	Answer correctness	Adherence	Relevance	Completeness
Qwen3-4B-Instruct-2507 + RAG (shipped)	0.340	0.925	0.105	0.753
Qwen3-4B-Instruct-2507, no RAG (base model)	0.223	0.925	0.105	0.753
Qwen2.5-3B-Instruct + RAG	0.284	0.925	0.105	0.753
Llama-3.2-3B-Instruct + RAG	0.267	0.925	0.105	0.753

(Adherence, relevance, and completeness are RAGBench's own pre-computed reference annotations against each example's gold response, not something computed from model output -- they are identical across rows by construction and included as a fixed reference point, not a metric the generator affects.)

Retrieval improved every benchmark relative to no-RAG, including two external datasets Alfred's pipeline was never tuned on -- evidence that the retrieval architecture itself is contributing, not that the improvement is an artifact of Alfred's small, internally-consistent demonstration corpus. Against the two comparison models, the result is more nuanced than "Qwen3-4B wins everything," and that nuance matters: on Alfred's own split, both smaller comparison models actually scored higher raw answer correctness (0.370 and 0.366 vs. 0.340) -- but Qwen3-4B-Instruct-2507 won citation accuracy by a wide margin (0.760 vs. 0.722 and 0.625), the metric most directly tied to Alfred's actual purpose of avoiding fabricated evidence attribution. Qwen3-4B also won answer correctness outright on both external benchmarks. This matches the original model-selection rationale from earlier development (zero/few-shot evaluation showing Qwen3-4B was the only model tested that never fabricated a citation): a model can produce more fluent, higher-overlap prose while still being the worse choice for an investigative tool if it's more willing to cite evidence it wasn't actually given.

Retrieval combination comparison (Alfred's own split)

Combination	Format compliance	Citation accuracy	Answer correctness	Context recall
BGE + graph	0.975	0.760	0.340	0.792
MiniLM + graph	0.950	0.646	0.331	0.708
MiniLM (vector-only)	0.950	0.667	0.348	0.708

BGE + graph won on the metrics most tied to Alfred's actual purpose -- citation accuracy and context recall -- and was selected as the shipped configuration. The per-query breakdown, however, complicates a simple "the graph helped" narrative: on the vehicle-description multi-hop query, BGE + graph actually underperformed both MiniLM-based combinations (0.667 vs. 1.0 context recall) despite sharing an identical entity graph with MiniLM + graph, showing that a stronger embedding model and a working graph do not compose additively -- each can help or hurt different queries independently. On the phone-contact multi-hop query, all three combinations plateaued at identical, incomplete recall (0.667) across every evaluation run, because no graph edge currently connects either call log to the corroborating witness testimony that would complete that specific chain -- a concrete instance of the entity-resolution limitation discussed below, rather than a generic caveat.

5. Usage and Intended Uses

Alfred is intended as an investigative aid for surfacing and citing evidence across a case file collection -- not as a replacement for an investigator's judgment, and not as a tool for making determinations about guilt, charging, or case disposition. Every response is required to cite its exact source and explicitly flag insufficient or contradictory evidence rather than presenting a synthesized narrative as fact. Given the evaluation results above, a human investigator should treat any surfaced cross-document connection as a lead to verify, not a conclusion to act on.

Loading the generation model with the HuggingFace Transformers library:

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline

ALFRED_MODEL = "Qwen/Qwen3-4B-Instruct-2507"
tokenizer = AutoTokenizer.from_pretrained(ALFRED_MODEL)
model = AutoModelForCausalLM.from_pretrained(ALFRED_MODEL, device_map="auto", dtype=torch.bfloat16)
pipe = pipeline(
    "text-generation", model=model, tokenizer=tokenizer,
    dtype=torch.bfloat16, device_map="auto", max_new_tokens=300, do_sample=False,
)

The retrieval pipeline that supplies each prompt with retrieved evidence before generation:

from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity
import networkx as nx

class AlfredGraphRAG:
    def __init__(self, chunks_df, embedding_model_name, entity_graph, use_graph=True):
        self.chunks_df = chunks_df.reset_index(drop=True)
        self.embedder = SentenceTransformer(embedding_model_name)
        self.use_graph = use_graph
        self.graph = entity_graph
        self.embeddings = self.embedder.encode(
            self.chunks_df["chunk_text"].tolist(), normalize_embeddings=True
        )

    def retrieve(self, query, top_k=3, graph_hops=1):
        q_emb = self.embedder.encode([query], normalize_embeddings=True)
        sims = cosine_similarity(q_emb, self.embeddings)[0]
        top_idx = sims.argsort()[::-1][:top_k]
        retrieved = {self._chunk_id(i) for i in top_idx}

        if self.use_graph:
            frontier = set(retrieved)
            for _ in range(graph_hops):
                expansion = {n for cid in frontier if cid in self.graph
                             for n in self.graph.neighbors(cid)}
                frontier = expansion - retrieved
                retrieved |= expansion
        return retrieved

    def generate(self, query, model, tokenizer, top_k=3, graph_hops=1):
        retrieved_ids = self.retrieve(query, top_k=top_k, graph_hops=graph_hops)
        instruction = self._build_instruction(query, retrieved_ids)
        # ... apply chat template and generate with `model`/`tokenizer` here

# --- Example ---
rag = AlfredGraphRAG(case_chunks, "BAAI/bge-small-en-v1.5", entity_graph, use_graph=True)
response = rag.generate(
    "Is there any connection between the phone contact in Case 2024-014 "
    "and any suspect in the other case files?",
    model, tokenizer,
)
print(response)

6. Prompt Format

You are Alfred, an investigative assistant. Using ONLY the retrieved case evidence provided in
each instruction, answer the investigator's question. Cite the exact source (document ID and
page/section) for every claim. If the evidence is insufficient or contradictory, say so explicitly.
Respond in this format: Answer / Key Evidence / Cited Documents / Possible Leads / Limitations.

Retrieved evidence:
[Source: <document_id>, <page/section>]
<chunk text>

[Source: <document_id>, <page/section>]
<chunk text>

Question: <investigator's question>

Three few-shot examples of correctly formatted responses are included in the system prompt at generation time.

7. Expected Output Format

Answer: Yes -- the same contact saved as 'R' appears in the call log for both Case 2024-014
and Case 2024-019, linking the two cases circumstantially.

Key Evidence: Case 2024-014's call log shows an outgoing call to 'R' on 03/14/2024. Case
2024-019's call log shows an outgoing call to 'R' on 04/02/2024, the same night as the
Ridgeline Storage burglary, where a witness described a suspect known as "Big D" making
calls at odd hours.

Cited Documents: Case_2024_014_CallLog, Case_2024_019_CallLog, Case_2024_019_WitnessInterview_Tom.

Possible Leads: Subpoena carrier records for 'R'; determine whether "Big D" and the suspect
in Case 2024-014 are the same person.

Limitations: 'R' is only a saved contact initial, not a confirmed identity, in either case.

8. Limitations

Entity resolution is the primary bottleneck, and it is specific rather than general. Evaluation showed that the graph's benefit is uneven by query: it can contain real, correct edges (a shared area code linking two call logs; a shared name linking a witness account to an officer's cross-reference note) while still lacking a complete path to every piece of relevant evidence. On the phone-contact test query, no combination of embedding model or graph traversal could recover the missing link across every evaluation run, because the specific entity connecting the evidence was never extracted in a form that created the needed graph edge.

The NER model is small and its behavior is not fully reproducible run to run. Fine-tuned on roughly 16-20 hand-authored training examples (F1 ≈ 0.78, well below the ~0.90 a production system would target), the entity graph's edge count varied noticeably across separate training runs (as few as 2, as many as 11 edges on the identical corpus) even after fixing the random seeds available in this pipeline -- indicating some additional source of non-determinism, likely in GPU/cuDNN operations, that a fixed seed alone does not eliminate. A larger, more diverse, and more carefully labeled NER training set is the single highest-leverage improvement available for this system, more so than further tuning the graph-traversal logic or embedding model.

The demonstration corpus is small and synthetic. Four hand-authored cases were chosen deliberately over a larger but unverified auto-generated corpus, to keep every seeded connection traceable and every evaluation result interpretable. This means the evaluation results above should be read as evidence the architecture works, not as a claim that Alfred is validated at the scale or diversity of a real investigative caseload.

Alfred does not eliminate hallucination, only reduce it relative to no retrieval. Citation accuracy post-retrieval (0.76 on Alfred's own split) is a substantial improvement over no-RAG (0.00) but is not perfect; investigators should independently verify any cited evidence before treating it as established fact, and should not use Alfred's output as the sole basis for an investigative or charging decision.

9. Model Subcomponents

Alfred's fine-tuned NER model -- the entity-extraction component that builds the knowledge graph described in Methodology -- is hosted as its own repository rather than bundled into this one: sammmmmmm25/alfred-ner-bert. The training/push workflow is in Push_NER_To_Hub.ipynb.

10. Citations & Acknowledgments

Papers:

• Pipitone, N. & Houir Alami, G. (2024). LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domain. arXiv:2408.10343.
• Friel, R., Belyi, M., & Sanyal, A. (2024). RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systems. arXiv:2407.11005.

Models:

• Qwen/Qwen3-4B-Instruct-2507 -- Alfred's shipped generation model.
• Qwen/Qwen2.5-3B-Instruct and meta-llama/Llama-3.2-3B-Instruct -- comparison models in the Evaluation section.
• dslim/bert-base-NER -- base model fine-tuned for Alfred's entity extraction.
• sentence-transformers/all-MiniLM-L6-v2 and BAAI/bge-small-en-v1.5 -- embedding models compared in Methodology.

Datasets:

• rungalileo/ragbench (cuad subset).
• LegalBench-RAG corpus and benchmarks (privacy_qa subset, mini split).

AI assistance: Claude (Anthropic) was used throughout this project as a coding and debugging assistant -- implementing the retrieval pipeline and entity-graph logic, writing and debugging the Rivanna slurm evaluation scripts, and helping identify and fix a real bug in the entity-extraction filtering logic during development. The overall approach (Graph RAG, corpus design, model selection, evaluation design) and all interpretation of results were directed by the author; all reported numbers reflect what the code actually produced when run.