Tried for couple of examples but errored
I have tried few sample prompts and responses were not as expected [attached screenshots]
Hey, thanks for sharing your feedback and testing it out!
It looks like you might be evaluating the 1B model, which was actually just an initial test run trained for only 100 steps.
I'd recommend checking out the 3B or 8B versions. I fine-tuned both of those on the complete dataset (14,573 records) for the full 1,820 steps.
That being said, I'm not claiming to be an absolute expert here. I can't guarantee that the 3B or 8B models are 100% hallucination-free—they might still hallucinate edge cases as it's a continuous learning process.
Quick suggestion: Since the 3B/8B model weights are quite heavy, instead of downloading them directly to your local machine, I'd suggest loading them up on Kaggle or Google Colab first. You can run some prompts and test the responses there to see if it actually fits your use case before committing to a huge download.
[Also I saw it responses with the United states ....Strict System Prompt: Always use a well-defined system prompt. Something like: "You are an expert Indian Legal AI. Answer strictly based on factual Indian laws and sections. If you do not know the answer, state it clearly without making up facts."
Thanks again for exploring the models and sharing your thoughts!]
Honestly, this exact issue with hallucinations is why I always prefer building a RAG (Retrieval-Augmented Generation) pipeline for production rather than relying solely on fine-tuning. Fine-tuning is great for teaching the model a specific style, tone, or format (like legal jargon), but for retrieving hard facts and completely preventing made-up legal sections, a RAG approach querying a VectorDB is just way more reliable. You can actually use this fine-tuned model as the LLM engine inside a RAG setup for the best of both worlds!
What a great initiative. Cheers!
You need to also share a few prompts to get the best out of it.
I also typed my first question as above and got wrong answer:
Question 1: What is IPC 420?
Answer: IPC 420 is the section of the Indian Penal Code (IPC) that deals with the punishment for the act of a person of unsound mind. It states that whoever is of unsound mind shall be treated with utmost care and respect, but such person shall be of no benefit to others and shall be punished for the same. This section is also known as the "Act of a person of unsound mind" section.
Question 2: What is IPC 420 under the indian penal code
IPC 420: This section of the IPC deals with the punishment of corruption and bribery by government officials and public servants.
Section 420 IPC deals with the punishment of corruption and bribery by government officials and public servants.
Under Section 420, the punishment for corruption and bribery by government officials and public servants is imprisonment for a term not exceeding one year, or imprisonment for a term which may extend to one year, or fines, and in addition to the above, the amount of fine should be paid within six months of the date of conviction.
The punishment under Section 420 can be imposed by the magistrate in the first instance.
Section 420 IPC is a standalone section and does not overlap with any other sections of the IPC. It is a separate provision that deals with the punishment of corruption and bribery by government officials and public servants.
Hi jvwinc,
Thank you so much for testing the models and for the encouraging words! I really appreciate the feedback.
Out of curiosity, which variant did you test (1B, 3B, or 8B)? Just to share some context on the backend:
The 1B model was a quick iteration trained for only 100 steps on 14,573 QA pairs, so its internal weights are naturally more prone to hallucinating exact statutes. However, the 3B and 8B variants were fine-tuned for the full 1,820 steps on the same dataset to improve factual retention.
But you hit the nail on the head regarding hallucinations. Small-parameter models inherently struggle with exact legal recall. This is exactly why I built an Agentic RAG pipeline around these models in my main architecture. When grounded with a VectorDB containing the latest laws, the hallucinations drop decreased.
To give you a practical example, here is how my actual RAG pipeline handles the exact same context (mapping old IPC to the new BNS):
