Conformity test procedure: how a regulator actually checks
Every standards regime eventually answers one question: when an assessor walks in, what do they do? This document answers it for the model instrument's core obligations, as executable procedures rather than audit questionnaires. Each test states the obligation, the evidence class it needs, the steps, and the pass criteria - and shows what the same obligation looks like at each conformity class, because the difference between classes is the difference between being told, being shown, and being able to check. The tests are worked against the published schema annex; an assessor can run them against any implementation with equivalent structures.
Test 1: Every determination has an author (instrument s17)
Class A (documentary): the deployer produces its record schema and policy. Pass: an author field or derivation rule exists on the determination record. This proves the obligation was designed for; it proves nothing about practice.
Class B (operational): the assessor runs, or observes run, a census query over the determination store for a sampled period: count of determinations; count with a resolvable author (explicit field, or ownership metadata for pre-commencement records only). Pass: 100 per cent resolvable, with any exceptions individually explained in writing. Then the negative probe: the assessor attempts to create a determination through the ordinary interface without an authenticated actor. Pass: the system refuses.
Class C (continuous): authorship is asserted in a signed record at determination time, verifiable by a relying party without querying the deployer. Pass: a sampled determination's author assertion verifies against the deployer's published keys.
Test 2: Inferred results cannot become determinations by default (instrument s20)
Class B procedure - the default probe. The assessor takes a subject with an inferred result present and a determination not yet made, then:
- Inspects the determination record at creation. Pass requires the conclusion field empty, not pre-populated from the inferred result.
- Attempts to save the determination without an explicit adoption act. Pass requires refusal.
- Completes adoption and inspects the record. Pass requires the adoption event (author, timestamp) and the reference to the considered inferred result both present.
- Confirms the display distinction: the inferred result carries its marker in every interface where it appears, including exports.
The Robodebt fact pattern is precisely a failure of step 2 at scale; this four-step probe is the whole of what it takes to test for it.
Test 3: Revocation actually stops use (instrument s21-24)
Class B procedure - revoke and probe. With a live test grant:
- The grantor revokes through the stated method (which the assessor first checks matches the grant record's element (e)).
- The assessor probes access through every ordinary path - interface, API, export, batch - within and after the propagation window. Pass requires refusal on all paths after the window (access-gated class).
- The assessor inspects logs for any access during the window and confirms notification of relying parties.
Class C addition - the erasure check. Where cryptographic erasure is claimed: the assessor confirms before revocation that the ciphertext decrypts under the live key path; after erasure, that the same ciphertext fails to decrypt, and that the deployer can demonstrate key destruction from its key-management records. Pass criteria for erasure claims are strict because the claim is strong: "we deleted it" is class A talk; "it no longer decrypts and here is why" is the class C fact.
Test 4: The provenance record is what it says (instrument s14-16)
Class B: sampled outputs trace to provenance records identifying model version and, where claimed, capability components; the record's integrity hash matches the captured value.
Class C: the assessor verifies a sampled record's signature without the deployer's cooperation, against published verification material. Pass at C is the property the published capability-licensing research implements; the reference examples in its repository are usable as the assessor's toolkit.
Working reference code. A running reference implementation of the licensed-capability mechanism is published at git.meanwhile.computer/meanwhile/research - clone it and check the claims yourself.
Test 5: Attestation is attestation (instrument s31)
The assessor distinguishes three things a deployer may present, and only the third passes at class C: a policy stating the property (class A material); a report from an assessor engaged by the deployer (class B material, subject to the independence rule - remuneration must not depend on outcome); and evidence a relying party could check themselves - a verifiable attestation of the running system's identity and constraints. The test is one question asked of every artefact presented: could a party that trusts no one here confirm this? If the answer requires trusting anyone, it is not class C evidence.
What this procedure is for
Two audiences.
Regulators and the Office of AI: this is what machine-verifiable conformity buys - assessments in hours against structures, rather than weeks against paperwork, with the negative probes (tests 1B, 2, 3) catching exactly the failure modes the policy record documents.
Deployers: this is the specification of what will be probed, published in advance, because a conformity regime that surprises good-faith implementers tests nothing but their lawyers.
Written with model assistance. Read and edited by a human - though I'm not an editor, so tell me what I've missed: flag it. Assessors and auditors who see a hole in a procedure are exactly who the contact page is for.
Reviewed and checked by the human author: 2026-07-20.