| --- |
| language: |
| - en |
| license: mit |
| base_model: microsoft/Phi-3.5-mini-instruct |
| tags: |
| - text-generation |
| - knowledge-graph |
| - information-extraction |
| - lora |
| - fine-tuned |
| - rdf |
| - nlp |
| - trustworthy-ai |
| - hallucination-detection |
| datasets: |
| - BSVGK/Text_to_KG_Construction_Dataset |
| metrics: |
| - f1 |
| - bertscore |
| pipeline_tag: text-generation |
| --- |
| |
| # Phi-3.5 Mini Instruct — Text-to-KG (UK Government Contracts) |
|
|
| ## Model Summary |
|
|
| This is a **LoRA fine-tuned version of Phi-3.5 Mini Instruct** trained to extract structured **RDF knowledge graph triples** from raw UK government procurement contract text. The model was developed as part of a UEL–Depixen industrial placement research project focused on building **trustworthy, hallucination-free domain-specific SLMs**. |
|
|
| ## Key Results |
|
|
| | Metric | Score | |
| |--------|-------| |
| | F1 Score | **0.9954** | |
| | BERTScore F1 | **0.9997** | |
| | Hallucination Rate | **0.00% (Zero)** | |
| | Test Contracts | 1,387 unseen contracts | |
|
|
| ## Model Details |
|
|
| - **Base Model:** microsoft/Phi-3.5-mini-instruct |
| - **Fine-tuning Method:** LoRA (Low-Rank Adaptation) |
| - **Task:** Text-to-KG — extracting RDF triples from contract text |
| - **Domain:** UK Government Procurement Contracts |
| - **Training Dataset:** 9,244 verified UK government contracts |
| - **Hardware:** NVIDIA A100 |
| - **Framework:** PyTorch, Hugging Face PEFT, TRL, SFTTrainer |
|
|
| ## Training Data |
|
|
| - **Source:** UK Government procurement contracts |
| - **Size:** 9,244 training samples | 1,387 test samples |
| - **Format:** Contract text → RDF triple extraction |
| - **Dataset:** [BSVGK/uk-contracts-text-to-kg](https://huggingface.co/datasets/BSVGK/Text_to_KG_Construction_Dataset) |
|
|
| ## Hallucination Evaluation Framework |
|
|
| This model was evaluated using a **novel dual-level hallucination evaluation framework**: |
|
|
| - **L1 — Relation Validity:** Checks if extracted relations exist in the ontology |
| - **L2 — Entity Grounding:** Verifies entities are grounded in the source contract text |
|
|
| This framework proved that **training loss alone is not a reliable quality signal** for KG extraction tasks. |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoTokenizer, AutoModelForCausalLM |
| |
| tokenizer = AutoTokenizer.from_pretrained("BSVGK/phi35-mini-lora-text2kg-merged") |
| model = AutoModelForCausalLM.from_pretrained("BSVGK/phi35-mini-lora-text2kg-merged") |
| |
| prompt = """Extract RDF triples from the following UK government contract text: |
| |
| Contract: [paste your contract text here] |
| |
| RDF Triples:""" |
| |
| inputs = tokenizer(prompt, return_tensors="pt") |
| outputs = model.generate(**inputs, max_new_tokens=256) |
| print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
| |
