Why I split my legal RAG into two models
While building a Retrieval-Augmented Generation system for legal document search, I hit a failure mode that has stuck with me ever since. The pipeline retrieved the right court documents and then summarised them — and roughly 3 in 10 outputs contained dates that looked completely realistic but did not appear anywhere in the source document. Filing dates, in particular, were being invented.
Why it happened
I had used BART for everything. BART is a generative, abstractive model: it predicts the most plausible next token. That is exactly what you want for a case summary, and exactly what you do not want for a filing date. A plausible-looking date is still wrong, and in a legal context "plausible but wrong" is worse than "I don't know."
The fix: route by query type
I stopped treating every question the same way. After retrieval (Sentence Transformers embeddings into a FAISS index, top-5 chunks by cosine similarity), I routed by what the question was asking for:
- Case summaries → BART, for fluent abstractive generation.
- Specific fields (plaintiff names, filing dates) → a BERT extractive QA model that can only return spans that actually exist in the document. It physically cannot hallucinate a date, because it is selecting text, not generating it.
That single change — extractive models for factual fields — essentially eliminated the invented-date problem, and dropped query time for counsel teams from minutes to under five seconds.
The lesson
Picking the right type of model matters as much as any hyperparameter. Generation and extraction are different tools; the trick was recognising that one question type needed each. The system ended up "hybrid" twice over: retrieval plus generation, and extractive plus abstractive.