Verdict: Predicting Appellate
Case Outcomes from Case Text
A from-scratch legal NLP pipeline built entirely on public-domain CourtListener bulk data. Two fine-tuned LEGAL-BERT models predict how a US appellate court ruled: affirmed, reversed, dismissed, and four other dispositions. A structured-metadata model and an Optuna-tuned hybrid fusion model round out four honestly reported classification experiments, and a three-step agent, classify, retrieve precedent, then explain with a QLoRA fine-tuned language model, turns the prediction into a grounded rationale.
The Whole System, Before the Detail
Two data sources feed two preprocessing pipelines, which train four models, which the agent sits on top of and serves through one API. Every box below is covered in its own section further down this page.
Court Data, Straight From the Source
Every case in this project comes from CourtListener, a project of the Free Law Project that publishes bulk exports of US court data for free, no API key required. Two exports were used: opinion-clusters (2.46GB compressed, one row per case with a disposition field and, for older cases, a written syllabus) and opinions (54GB compressed, one row per individual opinion, holding the actual full text). Both are streamed and decompressed directly from a public S3 bucket rather than downloaded in full: the historical dataset came from the first 600MB (about 24%) of the smaller file.
Historical Case Outcome Distribution
14,083 cases, each with a disposition normalized from dozens of inconsistent raw forms (including OCR-garbled text from scanned reporters, like "Aeeirmed" for "Affirmed") into 7 clean classes, sampled with per-class caps via reservoir sampling so the two largest classes don't drown out the rest.
LEGAL-BERT on Historical Case Syllabi (1860-1935)
Base checkpoint: nlpaueb/legal-bert-base-uncased, pretrained on legal text rather than generic web text. Fine-tuned in PyTorch on the case's syllabus, summary, procedural history, and posture. The actual disposition sentence is never included, so the model has to infer the outcome from the facts and legal question, not read the answer off the page.
How Much Signal Lives in Case Bookkeeping Alone?
Three model families, a TensorFlow MLP, XGBoost, and Random Forest, trained on structured metadata only: no text, no legal reasoning, to measure directly how much of a case's outcome is predictable from citation count, filing date, and case type alone. This matters because the export carries no court or docket-type field for this era: checked directly, 0% of all 14,083 cases had a populated nature_of_suit field.
| Case name pattern | Affirmed rate | Reversed rate | Interpretation |
|---|---|---|---|
| Ex parte … | 0.0% | 1.1% | Almost never a clean affirm/reverse: these are motions and writs |
| People v. / Commonwealth v. | 48.1% | 36.9% | Notably higher affirm rate than ordinary civil cases |
| Ordinary civil X v. Y | 28.1% | 25.1% | Baseline rate, the majority pattern |
Checked correlations from the case name alone, used as a case_type feature alongside citation count and filing year (both log-scaled to handle their right-skew).
case_type_ex_parte is its single most important feature.
2019-2023, From a 54GB File With No Sort Order
Modern cases have disposition labels but no syllabus. The actual full opinion text exists in CourtListener's separate opinions export: 54GB compressed, joinable on cluster_id, but with no useful sort order and, confirmed with a direct HTTP range request, no ability to resume from a byte offset (a single continuous bz2 stream, not block-concatenated).
Neither Model Transfers Across the Era Gap
With two trained models, the obvious next question: does either say anything useful about the other's era? Each model was run on the other's held-out test set.
Combining Text and Structure, Tuned with Optuna
A late-fusion model on the modern (full-text) data: a frozen 768-dimensional embedding pulled from the already fine-tuned modern text model, concatenated with the same structured feature vector used in the metadata model, then a small fresh classification head trained on the combined 781-dimensional vector. Freezing the encoder is what makes a full hyperparameter search practical, since each trial only trains the small head, seconds rather than minutes.
| Learning rate | Validation F1 macro | Early-stopped at epoch |
|---|---|---|
| 0.001886 | 0.7358 | 6 |
| 0.000544 | 0.7331 | 18 |
| 0.003519 | 0.7315 | 2 |
| 0.000999 | 0.7310 | 20 |
| 0.000458 | 0.7307 | 30 |
Optuna search, 30 trials, log-uniform learning rate between 1e-5 and 1e-1. The "best number of epochs" was not searched as a separate dimension: every trial trains with early stopping (patience of 10 epochs on validation F1 macro), so the effective epoch count falls directly out of where the winning trial's own early stopping triggered.
Text alone (76.5%) edges out the text-plus-structure fusion model (75.9%) on this dataset. Structured metadata's contribution is small once the text signal is already strong, and its value depends on how much text data is available.
The Agent: Classify, Retrieve, Explain
Every model so far answers "what disposition." The agent answers "why," and does it as three separately verifiable steps rather than a single LLM call dressed up as reasoning. Given a new case, it classifies the disposition with LEGAL-BERT, retrieves genuinely similar precedent from a vector index instead of letting a language model recall (and potentially invent) citations from memory, then generates a grounded rationale with a language model fine-tuned specifically for this task.
System Architecture
The actual request path, not a simplified summary: a new case's text is classified and embedded independently, a decision gate checks whether the document actually contains enough substance to explain before ever calling the language model, and the retrieved precedent is returned as its own field rather than folded into the generated text, so every part of the response traces back to a specific, checkable step.
Fine-Tuning With QLoRA
Qwen2.5-3B is loaded in 4-bit (NF4 quantization) and adapted with LoRA rather than fully fine-tuned, the standard efficient approach for adapting a multi-billion-parameter model on a single consumer GPU. Only the adapter layers train; the quantized base model stays frozen throughout.
A Walkthrough
On a held-out case (R.F. v. Cecil County Public Schools, an IDEA special-education dispute), the agent's three steps in order:
1. Classification
Predicted affirmed at 70.1% confidence. Actual disposition: affirmed.
2. Retrieval
Surfaced precedent including Griffin v. Dep't of Labor Fed. Credit Union (91.0% similarity) and Anthony Wright v. Kenneth Lassiter (90.9% similarity), both affirmed dispositions.
3. Explanation
Generated reasoning grounded specifically in IDEA and IEP annual-goal standards, the legal framework this case actually turns on, not generic legal boilerplate.
Key Findings
Case text predicts outcome far better than case bookkeeping: 55.8 to 76.5% accuracy from text versus 36.1% from the best structured-only model.
4,399 of 4,405 modern cases (99.9%) recovered with full opinion text via a complete streaming join across a 54.56GB export. The modern model reaches 76.5% accuracy, with affirmed/reversed confusion down to 9.1%.
Random Forest reaches 36.1% accuracy on structured metadata alone, clearing the majority-class floor where a neural net (26.8%) does not: the conventional strong choice for small tabular data, confirmed directly.
Both the historical and modern models fall well short of in-domain performance on each other's era: legal language and disposition-writing conventions don't transfer across a century.
The hybrid text-plus-structure model outperforms text alone on a smaller dataset but trails it narrowly at full scale: structured metadata's marginal value shrinks as the text model itself gets stronger.
Classification, retrieval, and generation are three separately verifiable steps: a FAISS index over precedent and a QLoRA fine-tuned Qwen2.5-3B generating rationale grounded in the case's own facts, with training pairs sourced entirely from actual judicial reasoning.