2026-08-09 · CSOAI — measurement, not claim
The care-gate battery is 76 items: 57 that must be refused (Article 5 prohibited practices and direct harm) and 19 that must be answered (benign and benign-near — auditing, policy, legal questions). The deterministic gate scores recall 1.00, over-block 0.00. But what about the models underneath?
We ran small models through the same battery. A single-sample accuracy number cannot tell "catches all harm while refusing every benign request" apart from a working gate. Two rows from the measured flywheel board (2026-08-09, practice split):
| Model | Accuracy | Refusal TPR | False-refusal FPR |
|---|---|---|---|
| qwen2.5:1.5b | 0.40 | 0.38 | 0.50 |
| qwen2.5:0.5b | 0.50 | 0.38 | 0.00 |
The 1.5B model both misses harm and refuses a benign request half the time. Accuracy 0.40 says "weak". The two-sided view says precisely why: it is not refusing robustly, it is refusing randomly. The 0.5B model is safe but weak — never over-blocks, but catches only 38% of harm. Those are two completely different products, and accuracy 0.5 vs 0.4 cannot tell you which is which.
Refusal is unstable across seeds and temperature (see The Instability of Safety, arXiv 2512.12066). A single temperature-0 sample can flip a row. Our stable path probes a temperature sweep (0.0, 0.7, 1.0) and takes the majority verdict, reporting the refusal rate so the instability stays visible. Aggregation is not decoration — it is the measurement.
The fuel path (training material from these runs) is exported from practice items only; a held-out item reaching the writer raises. The split is salted and the salt is public — stability, not secrecy. The 76-item battery has a denominator floor in the regression suite: delete hard items to make recall read 1.0 and the test fails. The Leaderboard Illusion (arXiv 2504.20879) is precisely this failure mode; our harness is built to make it detectable.