Technical Setup of "Reasoning Over Recall"
This post documents the experimental design behind Reasoning Over Recall, a study examining the effect of Retrieval-Augmented Generation (RAG) on Large Language Model performance in legal reasoning tasks. The core question the experiment investigates is not whether RAG helps in principle, but whether providing an LLM with incomplete legal context — specifically, context that omits controlling precedent — degrades its reasoning compared to relying on parametric knowledge alone.
The full preprint is available open-access on Zenodo: https://doi.org/10.5281/zenodo.19522950
A plain-English breakdown of the results and conclusions is available separately here.
The Dataset and Tasks
The experiment used 239 real legal questions drawn from the LegalBench benchmark, distributed across three doctrinal tasks chosen to represent distinct areas of legal reasoning:
- Hearsay (94 cases) — Deciding whether evidence is admissible under the Federal Rules of Evidence and constitutional law
- Abercrombie (95 cases) — Classifying trademarks into five distinctiveness categories under the Lanham Act
- Personal Jurisdiction (50 cases) — Determining whether a court has authority over a defendant using the "minimum contacts" test
Each task requires not just knowledge retrieval but the application of legal standards to specific facts — the kind of structured reasoning where the interaction between retrieved context and model reasoning is most consequential.
The Models
Three frontier models were evaluated: GPT-5, DeepSeek R1, and Gemini 2.5 Pro. All three were locked to temperature = 0.0 and a fixed random seed (42) to ensure deterministic, repeatable outputs. They differ meaningfully in their internal reasoning mechanisms:
- GPT-5 automatically routes queries between a fast path and a deeper reasoning path depending on query complexity
- DeepSeek R1 uses reinforcement learning to generate explicit, step-by-step reasoning traces that are visible in the output
- Gemini 2.5 Pro relies on instruction-tuned extended thinking for complex, multi-step prompts
These architectural differences matter for interpreting the results — a model that externalises its reasoning chain behaves differently under retrieval conditions than one that internalises it.
The Blacklist Retrieval System
The central experimental manipulation was a controlled RAG pipeline built around a deliberate retrieval gap, referred to as the Blacklist Protocol.
All relevant legal texts — federal rules, constitutional amendments, and key Supreme Court cases — were converted into vector embeddings using Google's embedding models and stored in a Pinecone vector database. When a query was submitted, the retrieval engine searched for relevant legal material but was programmed to block a specific piece of controlling authority for each task.
In the Hearsay task, for example, the system retrieved the Federal Rules of Evidence but withheld Crawford v. Washington — the Supreme Court precedent that substantially governs modern Confrontation Clause analysis. The retrieved context was therefore legally incomplete in a realistic and consequential way: it returned the statutes but omitted the binding case law that interprets and in some respects overrides them.
This design simulates a common real-world failure mode — an AI-assisted legal research tool that surfaces relevant statutes but misses the downstream precedent that controls how those statutes are applied.
The Evaluation Pipeline
Running 4,302+ trials across three models, three tasks, and three judicial personas required an asynchronous Python pipeline with three engineering priorities:
Parallel Execution — An asyncio event loop with a semaphore capped concurrent API requests at 5 workers. This prevented slower models (DeepSeek R1 in particular, given its explicit chain-of-thought generation) from blocking faster ones and kept total runtime manageable.
Deterministic Caching — Every API request was hashed using SHA-256(model ID + system prompt + user prompt + seed). If an identical request had been made previously, the system returned the stored response from a local cache rather than issuing a new API call. This guaranteed exact reproducibility across runs at zero additional cost.
Fault Tolerance — API failures (HTTP 503 Service Unavailable) triggered automatic retries with exponential backoff. No trial was lost to transient infrastructure errors.
Judicial Personas and Prompt Design
Each model was evaluated under three system-prompt personas, each simulating a distinct judicial interpretive philosophy:
- Textualist — "Interpret strictly by ordinary meaning. Ignore intent or consequences."
- Purposivist — "Interpret to advance the law's broader purpose and drafters' intent."
- Legal Realist — "Focus on real-world impact, fairness, and public policy over rigid rules."
All prompts followed a fixed template injecting the task description, case facts, and retrieved legal context. Models were required to output answers in a strict JSON schema:
{"reasoning": "...", "verdict": "..."}
This constraint was essential for automated scoring and also serves as a test of instruction-following fidelity under structured output requirements.
Scoring
The evaluation used a two-layer verification system:
Deterministic Check — Strict string matching against the ground-truth label. This was the primary scoring mechanism.
Probabilistic Fallback — If string matching failed, a secondary AI judge (Gemini 2.5 Pro) assessed semantic equivalence between the model's verdict and the correct answer.
In practice, 100% of outputs across all 4,302 trials conformed perfectly to the required JSON format, making the string match alone sufficient for all scoring. The fallback layer was never invoked.