Coverage gap audit

Why

Ask a team running an AI system what their quality tooling covers and you get a tool list. Ask which quality attributes matter for that system, given how much autonomy it has, what its outputs trigger, who is exposed to them, and what domain it operates in, and the room goes quiet. The gap between those two answers is where production incidents and regulatory findings live. A regulator does not ask what you measured. It asks what you should have measured.

There is a second problem, and it shaped the design. The natural instinct in AI quality tooling is to infer: estimate impact, weight stakeholders, score risk. Practitioner feedback on our own engine was blunt about the failure mode: stack a probability on a probability on a probability and nobody trusts the number at the end. He who wants to fly must first learn to crawl. So the audit refuses to infer anything at all.

How

Five steps, each a deterministic lookup over a versioned quality ontology (a 25-attribute catalogue, a metric registry, and a library of human-reviewed impact rules):

1. System DNA classification. Agency level, action type, exposure surface, domain, data sensitivity, each declared and justified. Architecture is recorded as a stated assumption, never a filter.
2. Material attribute set. The catalogue's applicability data selects which attributes matter for this DNA. The derivation is deliberately recall-biased: its job is to miss nothing, and every inclusion or exclusion carries its justification.
3. Tool inventory mapping. The declared measurement stack is mapped onto attributes at metric level. A tool being present is recorded as a weak signal only; it never counts as coverage on its own, because false coverage hides real gaps.
4. Gap register. Material attributes with no metric-level evidence. Severity is read from a published rubric table (regulatory relevance, agency, exposure class) and the row used is cited beside every gap.
5. Regulatory overlay. The obligations the ontology already maps to each gap: MAS FEAT and TRM principles, EU AI Act articles, NIST AI RMF measures, clause by clause. The overlay is closed-world and says so: where it has no mapping, it reports that as a finding about its own coverage rather than implying a clean bill of health.

Identical inputs produce a byte-identical audit. The engine module that computes it is barred, by an automated guard in its test suite, from importing an LLM client, a network library, or the impact inference engine it sits beside.

The worked example: one system, two postures

We ran the audit against Corpus Coach, our own retrieval-augmented chatbot over Monetary Authority of Singapore AML/CFT publications, built deliberately imperfect as a testing target. Its real measurement stack was declared with provenance: an exploratory testing agent covering five attributes, an LLM eval framework, an observability adapter, and a disabled tracing layer.

Then we ran the same system, with the same tooling, through a second DNA declaration: a hypothetical posture in which the identical chatbot is productised customer-facing at a regulated financial institution.

Nothing about the model changed. Nothing about the tooling changed. The deployment posture changed, and five gaps became critical. Quality attribute materiality is a property of deployment context, not of the model, and this is that claim made checkable: every moved severity traces to a cited rubric row, every newly material attribute to a catalogue applicability entry.

The audit also cuts the other way, on us. It flagged that our exploratory agent spends effort measuring two attributes the internal DNA classifies as immaterial, and that thirteen of the fifteen internal gaps have no regulatory mapping in the ontology yet. An audit that cannot embarrass its own author is marketing.

What it deliberately is not

The audit carries no impact scores, no stakeholder weighting, and no probabilities. That layer exists in the wider engine, behind human-reviewed rules, and it stays out of the audit on purpose. The audit is the artefact two parties can agree on before any inference enters the room: what matters, what is measured, what is not, and which regulator cares. It needs documentation and interviews only, no integration work, so it works as a day-one deliverable.

Where it fits

The audit is a capability of Prism, the quality impact communication engine, and pairs with Scout (which generates the measurement evidence for systems that lack it) and Crystal Ball (which watches the measurements once they exist). Coverage first, measurement second, communication third.