Reasoning Over Recall
The standard prescription for deploying AI on knowledge-intensive tasks is Retrieval-Augmented Generation: rather than relying on what a model learned during training, you give it a search tool. For legal questions, this means retrieving the relevant statute and feeding it to the model before it answers. The intuition is sound — surely providing the actual text of the law makes reasoning more grounded and accurate.
My recent research suggests that in the legal domain, this assumption does not hold under realistic retrieval conditions. Specifically, I found that providing an AI model with a statute while withholding the controlling case law that interprets it can degrade performance relative to providing no retrieved context at all.
The complete research paper is available open-access on Zenodo: https://doi.org/10.5281/zenodo.19522950
The Experimental Design
The study evaluated three frontier models — GPT-5, DeepSeek R1, and Gemini 2.5 Pro — across 4,302 experimental trials drawn from 239 legal cases in the LegalBench benchmark. The tasks covered three areas of doctrine: Hearsay admissibility, trademark distinctiveness under the Abercrombie framework, and Personal Jurisdiction under the minimum contacts test.
Each model was evaluated under two conditions:
- No-RAG (Baseline) — Models answered using only their pre-trained parametric knowledge, with no retrieved context provided
- Statute-Only RAG — A "Blacklist Protocol" allowed the retrieval system to surface the governing statute but programmatically withheld the controlling Supreme Court precedent that modifies it
This design isolates a specific and common real-world failure mode: a legal research tool that surfaces relevant statutes but misses the downstream case law that interprets, narrows, or constitutionally supersedes them.
The Finding: Statutory Myopia
Across all three tasks, models performed worse when given the statute than when given nothing at all. Ensemble accuracy fell from 87% in the No-RAG baseline to 78% in the Statute-Only RAG condition — a degradation of 9 percentage points attributable entirely to the presence of incomplete retrieved context.
I term this pattern Statutory Myopia: the tendency of AI models to fixate on explicit text placed in their context window, even when that text contradicts broader controlling precedents already present in their parametric memory. The retrieved content does not supplement the model's knowledge — it overwrites it.
The Crawford Trap
The effect was most pronounced in the Hearsay task, where accuracy dropped by 21 percentage points. The mechanism is traceable to a specific legal structure I call the Crawford Trap.
Under Federal Rule of Evidence 803, certain hearsay statements are admissible. However, the Supreme Court's decision in Crawford v. Washington imposes a constitutional constraint that operates at a higher level of the legal hierarchy: if admitting a statement would violate the Sixth Amendment's Confrontation Clause, it cannot be admitted regardless of what the statute permits. The case law supersedes the statute.
When operating in the No-RAG condition, models correctly retrieved Crawford from training memory and applied its constitutional bar. When the Statute-Only RAG condition placed FRE 803 in the context window, models developed tunnel vision — they applied the permissive statutory rule in front of them and disregarded the prohibitive constitutional rule they already knew. Explicit text displaced implicit knowledge.
The Persona Gap
The models' vulnerability to Statutory Myopia was not uniform — it varied significantly with judicial persona. Models prompted to reason as Textualists (interpreting strictly by the written word) showed the largest performance degradation, up to 12.1%, because they were primed to privilege whatever explicit text appeared in the context window. Models prompted as Legal Realists (reasoning from real-world impact and broader principles) were considerably more robust, declining by as little as 3.8%, because they were more willing to look past the immediate retrieved text and draw on wider legal principles.
This finding has a practical implication: the framing given to an AI legal assistant shapes not only its interpretive style but its susceptibility to being misled by incomplete retrieval.
Implications for Legal AI Systems
The findings challenge a widely held assumption in current legal AI development. Providing retrieved context is not a neutral operation — in hierarchical domains like law, partial context actively distorts reasoning by outcompeting the model's own parametric knowledge.
For a RAG-based legal assistant to be reliable, it cannot retrieve statutes in isolation. It requires an understanding of the structure of the law — specifically, the hierarchical relationships between statutes, regulations, and the precedents that interpret and supersede them. One viable architectural direction is graph-structured knowledge bases that automatically link a retrieved statute to its authoritative downstream case law, preventing the retrieval system from serving a statutory provision without the constitutional and precedential constraints that govern its application.
The core finding is precise: in legal reasoning, a little retrieved knowledge can be a more dangerous thing than none at all.