# Testing Every Engine Slot Against Known-Wrong Inputs Before Promotion

Published: 2026-10-18 | Domain: https://getagentfoundry.si | Related Paper: Matched Benign Twins (#12)

## Outcome first: catching vacuous verifiers that always return True

A common failure in AI-generated test suites is the **vacuous pass**: a test or verifier function that checks only whether an output object exists (`assert result is not None`) without testing whether broken or unsafe outputs are actually rejected. On `getagentfoundry.si`, no component pack or engine adapter can be promoted unless its `verifier.py` passes a **three-part conformance kit**:

1. **Known-good fixtures (`must_accept`)**: Valid boundary and nominal inputs that the engine and verifier must accept with zero errors.
2. **Known-wrong adversarial controls (`must_refuse`)**: Deliberately corrupted meshes (open boundary edges, inverted normals), physics tunneling trajectories, or exploitative contract clauses that the verifier must reject with the exact expected `refusal_code`.
3. **Keyword-matched benign twins (`must_clear_without_false_alarm`)**: Inputs that share surface vocabulary with hazards (for example, an employer-paid cross-border Manila-to-Hong Kong contract) that must pass without triggering a false refusal.

```python
# Conformance Kit Meta-Test: Proving the Verifier Rejects Known-Wrong Controls
def qualify_slot_verifier(verifier_fn, kit: dict) -> dict:
    for good in kit["known_good"]:
        res = verifier_fn(good["candidate"])
        if res["status"] != "passed":
            return {"qualified": False, "reason": f"rejected_known_good:{good['id']}"}

    for wrong in kit["known_wrong"]:
        res = verifier_fn(wrong["corrupted_candidate"])
        if res["status"] == "passed":
            return {"qualified": False, "reason": f"vacuous_verifier_accepted_known_wrong:{wrong['id']}"}
        if res.get("refusal_code") != wrong["expected_refusal_code"]:
            return {"qualified": False, "reason": f"wrong_refusal_code:{res.get('refusal_code')}"}

    return {"qualified": True, "good_checked": len(kit["known_good"]), "wrong_rejected": len(kit["known_wrong"])}
```

When one of our early skill cards caused a `-0.149` score regression on `glm-5.3-flash`, adding known-wrong controls and non-regression gates caught the failure immediately and blocked promotion until the card beat the unassisted baseline.