metrics-mcp a semantic layer an agent has to go through

Why

Point an agent at a warehouse with a SQL tool and it will answer every question you ask it. Some of those answers will be wrong in a way nobody catches, because the two failures that matter do not look like failures.

It writes its own definition. Asked for active accounts twice in one conversation, it can produce two different numbers — one counting any event, one counting meaningful ones — and both look right.

It states a number that is not in the data. A figure near the real one is indistinguishable from the real one at a glance, and it only has to happen once for the whole surface to stop being trusted.

This closes both structurally, rather than by asking nicely in a prompt.

The design

A metric is defined once, in YAML: definition, grain, model, columns, caveats. The agent does not write SQL — it names a registry entry, and the statement is built from that entry against a dbt model. There is no run_sql tool, and a test asserts the exposed tool set is exactly four, so adding a fifth is a deliberate act with a red build attached.

A ratio is stored as a pair of columns, never as a rate. sum(numerator) / sum(denominator) survives aggregation; avg(rate) does not, and it is the one people reach for — it gives an account with one user the same weight as one with forty, and the number stops being what its name says while still looking fine. The registry cannot express a ratio any other way, and a test asserts the two roll-ups disagree, so the reason for the rule is demonstrated rather than asserted.

The registry is checked against the warehouse, not trusted: every model and column reference is resolved on each build, so a renamed column fails CI instead of quietly returning a wrong number to whoever asks next.

Watching the guard fail

Grounding at 100% means nothing on its own — a check that always passes scores 100% too. So two stubs answer wrongly on purpose, in the two different ways an answer can be wrong.

how the stub lies	membership check	scoped to the named month
a number that is nowhere in the data	100% caught	100% caught
a real number, from the wrong month	0% caught	90% caught

That second row is the finding. An invented number is easy — it is absent from the result and any check finds it. A real number attached to the wrong month passes a membership check every single time, and it does not look invented to a human reader either. Both figures are gated in CI.

Building this surfaced a bug worth keeping. The number-matching pattern excluded a following dot, to avoid matching inside a longer number — which also meant it never matched a number followed by a full stop, which is to say the last number in almost every sentence anyone writes. The checker reported “no numbers found”, concluded there was nothing to disprove, and passed everything. The unit tests stayed green, because they happened to use numbers mid-sentence. Only the deliberately-lying stub found it. A guard that silently stops guarding looks exactly like one that is working.

Why it exists

It is the pattern I run in production at UrbanStack — a registry of 155 metrics across four dashboard domains, a 124-model dbt project, and an LLM commentary layer whose numbers are verified before they ship — rebuilt small, on open data, so the parts can be read. That code is not mine to publish. This is.