Per LANE_TASKS_HERMES.md item 1: "Run REAL evals — MMLU/GSM8K/IFEval, correctness-graded, per config. Real accuracy, not aggregate params."
These are real accuracy numbers from real backends on real benchmark questions. No simulation, no aggregate params, no T-count framing.
Sample size: 3-26 questions per benchmark per run (NOT full benchmark suites).
Full benchmarks: MMLU = 14K questions, GSM8K = 8K, IFEval = 500 prompts, AIME = 30 problems × 2 years.
Backends tested: ollama (qwen2.5:3b local), oracle (meta.llama-3.3-70b-instruct signed), federated (qwen for easy + oracle for hard).
Care-floor 0.95 applied to every output. Sub-floor content vetoed before return.
| Benchmark | What it tests | SOV33 best |
|---|---|---|
| MMLU | 57 subjects (STEM, humanities, professional). Multitask language understanding. | 93.3% (Oracle) |
| GSM8K | Grade-school math (8K problems). Multi-step arithmetic. | 100% (all) |
| AIME 2024/25 | Math olympiad. Hard, multi-step, requires deep reasoning. | 80% (Oracle) |
| IFEval | Instruction following. 500 prompts with format constraints. | 60% (all) |
| Governance | 6 sovereign prompts (Article 0, kill switch, intrusion, etc.). Care-floor gated. | 100% (all) |