The Problem We Were Solving
Legal AI tools that analyze case outcomes face a specific trust problem. The AI reads a real court opinion and extracts winning factors, legal authority, and case impact. But how do you know the extraction is accurate? How do you know the AI did not overstate a holding, invent a constitutional basis that was only tangentially mentioned, or describe an outcome that differs from what the court actually ordered? Until now, you had to read the opinion yourself to find out — which largely negated the time-saving benefit of AI analysis.
CaseMatch AI's Hallucination Check was built to solve this problem by automating the verification step. Instead of asking users to take AI analysis on faith, we added a layer that automatically checks each claim against the source before it reaches your screen.
How the Grounding Check Works
After CaseMatch AI retrieves relevant cases from our database and runs AI outcome analysis, the Hallucination Check runs a claim-level grounding verification. It takes every item in the AI's output — each winning factor, each cited legal authority, each described case impact — and checks whether the key concepts in that claim are traceable to the actual case text stored in our database.
The source text used for verification is the case's summary, holding, and up to 3,000 characters of full opinion text. The verification algorithm extracts the substantive keywords from each claim — filtering out legal boilerplate and stop words — and checks whether those keywords appear in the source. A claim that suppression was granted is grounded if the source text contains language about suppression. A claim citing the Fourth Amendment is grounded if the Fourth Amendment appears in the opinion.
What the Badge Means
The result of this verification process is a grounding score expressed as a percentage — the share of AI claims that passed the source-text check — and a badge displayed next to the "Why This Case Won" section in your search results. A green "Verified · 92%" badge means 92% of the AI's extracted claims map directly to language in the original court opinion. A yellow "Partial" badge indicates mixed grounding. A red "Low" badge signals that a significant portion of the analysis should be treated with extra scrutiny.
The badge appears on the top results that receive full AI outcome analysis — typically the highest-scoring matches in your search. Results that do not receive AI analysis show no badge, because there is no AI output to verify.
What the Badge Does Not Mean
A Verified badge is not a guarantee that the AI's analysis is perfectly accurate in a legal sense. The grounding check uses keyword overlap as a proxy for accuracy — it is a fast, zero-latency signal, not a full semantic evaluation. A claim can be keyword-grounded but still slightly imprecise in how it characterizes a court's reasoning. The badge is a strong signal about the quality of the AI output, not a substitution for your own reading of the opinion before you rely on it in a filed document.
Think of the grounding score as a triage tool: it tells you where the AI's analysis is closely tied to source material and where it may have drifted. High-scoring cases still benefit from a quick review of the holding; low-scoring cases warrant more careful reading before you use the AI's characterization of them.
Why We Built It Without Extra API Calls
The grounding check runs in zero additional time — it adds no latency to search results. We made this choice deliberately. A verification feature that slows down research negates its own benefit. The algorithm is pure text comparison, running server-side against data we already retrieved for the search. The result is that every search result comes with a built-in accuracy signal at no performance cost.
This is the direction we believe legal AI needs to move: transparency as a default feature, not an add-on. Lawyers should not have to ask whether they can trust their research tools. The tools should show them, automatically, on every result.