lease-validator evidence or nothing

The problem

A compliance check is only useful if you can see why it decided what it decided. A model that returns “this lease violates the deposit cap” and nothing else has moved the work, not done it — somebody still has to open the document and find the clause.

So the rule here is absolute: every pass or fail carries a verbatim quote, and the quote is verified against the source before the verdict is returned. A verdict whose quote does not survive that check is demoted to unclear rather than shipped. The system is allowed to be uncertain. It is not allowed to be unaccountable.

How it works

Rules are YAML files, not code. Each one is judged by its own retrieval-grounded agent — one conversation per rule, not one pass over the document. Six rules in a single prompt means six sets of guidance competing, and the failure is quiet: four answered well, two phoned in. Per-rule runs also mean each rule gets the passages its own queries found, its cost is attributable, and a rule that goes wrong can be re-run alone.

The agent has three tools: search the document, step to neighbouring passages, and submit a verdict as a structured tool call so the schema is enforced by the API rather than by parsing. Retrieval is heading-aware, because in a lease the unit of meaning is the clause and its heading is often the only thing that disambiguates it — a blind 1000-character window cuts “not less than sixty (60)” away from “days’ written notice”.

What the evaluation says

30 generated leases × 6 rules = 180 labelled pairs, balanced so each rule has an equal number of pass, fail and unclear cases.

	keyword baseline	model
three-class accuracy	75.6%	97.2%
violation F1	0.692	0.976
violation recall	61.7%	100%
F1, hard tier	0.061	0.960

The hard tier is the row that matters. Those are documents where the answer is deliberately hidden — a defined term whose value lives pages away, a fee that only means something next to the rent, a larger decoy number sitting in the same section. The baseline scores 0.061 there, which is the honest way of saying it cannot do this at all. The gap between the two columns is the whole argument for the retrieval and the agent loop; without the baseline the 0.976 would just be a number.

Every error is attributed to a layer — retrieval or reasoning — so a bad result points at the thing that has to change.

The parts people skip

Silence is unclear, never pass: a document that simply does not mention a requirement has not satisfied it. The run refuses to score itself if it did not really happen — a crashed or truncated run reports as failed rather than as a good score on fewer cases. And the whole test suite (109 tests) runs offline, with no key and no network, so CI can actually gate on it.