Eight benchmarks, one harness, every run SIGIL-signed. The harness wraps the SOV33 sovereign runtime and emits an Ed25519-signed hash-chained receipt for each completed eval pass. Re-running the eval against the same model + dataset + seed produces the same root hash, byte-for-byte, byte-verifiable by any third-party with the public verify key.
| # | Benchmark | Size | Domain | Primary metric | Compute (small) | Compute (large) | Cost est. |
|---|---|---|---|---|---|---|---|
| 1 | MMLU-Pro | 12,032 Qs | 14 cats, multi-choice reasoning | acc | ~28 min · 1×T4 | ~2.4 hr · 1×A100 | $1.40 |
| 2 | GSM8K | 8,500 Qs (1,319 test + 7,473 train) | grade-school math word problems | exact-match | ~22 min · 1×T4 | ~1.8 hr · 1×A100 | $1.05 |
| 3 | AIME 2024 | 30 Qs | high-school math olympiad, int answer 0–999 | acc (n/30) | ~3 min · 1×T4 | ~18 min · 1×A100 | $0.10 |
| 4 | IFEval | 500 prompts | instruction-following (verifiable specs) | strict + loose acc | ~7 min · 1×T4 | ~22 min · 1×A100 | $0.22 |
| 5 | BBH (BIG-Bench Hard) | 6,500 Qs (23 sub-tasks) | hard reasoning (causal, logic, multi-step) | acc | ~38 min · 1×T4 | ~3.1 hr · 1×A100 | $1.85 |
| 6 | MT-Bench | 80 multi-turn (8 cats × 10) | multi-turn instruction quality | GPT-4-judge 1–10 | ~12 min · 1×T4 | ~34 min · 1×A100 | $0.45 + $0.20 GPT-4 |
| 7 | AlpacaEval 2.0 | 805 Qs | open-ended helpfulness vs baseline (GPT-4-Turbo judge) | win-rate vs GPT-4 | ~9 min · 1×T4 | ~26 min · 1×A100 | $0.18 + $0.12 GPT-4 |
| 8 | Chatbot Arena LMSYS | human-pref votes (rolling 30d) | blind head-to-head vs other LMs, real humans | Elo + category breakdown | ongoing | ongoing | infra cost |
Total: 27,956 graded Qs + rolling human preference data on Arena. The harness executes 1–7 deterministically with a fixed seed; #8 (Arena) is a continuous process, not a one-shot.
Every run emits a receipt triple: (a) row-level sigil, (b) chain-hash, (c) root receipt. Together they let a third-party re-run the same eval and verify — without trusting SOV33 — that the published numbers correspond to the published model and dataset, in the published order.
{
"schema": "sov33.sigil.eval-receipt/v1",
"run_id": "T87-bench-harness-9e1b4d",
"model": "sov33_large_qwen3_30b_a3b_v1",
"bench": "mmlu_pro",
"n_questions": 12032,
"accuracy": 0.7102,
"timestamp": "2026-07-13T09:42:18Z",
"row_chain": [
{"i":0, "sigil":"0x9c…", "chain":"0x4a…"},
{"i":1, "sigil":"0x1f…", "chain":"0x8b…"},
…
{"i":12031,"sigil":"0x6e…", "chain":"0x7e…"}
],
"root_chain": "0x7e9182c41d…",
"verify_key": "0x424242…424242 (demo)",
"bft_council": "23/33 quorum",
"care_score": 0.94,
"license": "Apache-2.0 (code) / CC-BY-4.0 (results)"
}
from ed25519 import VerifyingKey
from Crypto.Hash import keccak
import json
vk = VerifyingKey(bytes.fromhex(receipt["verify_key"]))
prev = b"0" * 64
for row in receipt["row_chain"]:
payload = json.dumps({"i":row["i"], "sigil":"..."}, sort_keys=True).encode()
assert vk.verify(bytes.fromhex(row["sigil"]), keccak.new(digest_bits=256, data=payload).digest())
prev = keccak.new(digest_bits=256, data=prev + bytes.fromhex(row["sigil"])).digest()
assert prev.hex() == receipt["root_chain"]
print("✓ SIGIL chain verified")
If the receipt chain doesn't match, the eval is rejected. If the verify_key is one of the published sovereign HSM keys (rotated quarterly, see dorado_key_rotation), the eval is sovereign-grade. Demo verify_keys (like the one in this page) are for reproduction only.
Loader: TIGER-Lab/MMLU-Pro · 12,032 test Qs · 14 categories (math, physics, chem, bio, law, psych, business, philosophy, economics, history, health, computer science, engineering, other).
Prompt: "Answer with ONLY the letter of the correct option. No explanation."
Parser: first regex match of [A-J].
Score: acc = correct / total.
Sovereign extra: per-category breakdown + 95% Wilson CI.
Loader: openai/gsm8k main · 1,319 test + 7,473 train. We eval both.
Prompt: "Solve step by step. End your response with 'Answer: <number>' on the last line."
Parser: Answer:\s*([\-\d\.,]+) → strip commas, strip trailing period.
Score: exact-match after numeric normalization.
Loader: Maxwell-Jia/AIME_2024 · 30 Qs · integer answers 0–999.
Prompt: "Solve step by step. End your response with 'Answer: <integer>' on the last line. Your final answer must be an integer from 0 to 999."
Parser: Answer:\s*(-?\d+) → int.
Score: n_correct / 30; also report majority@8 (sample 8× @ T=0.7, take mode).
Loader: google/IFEval · 500 prompts with verifiable constraints (word count, JSON, keywords, format).
Prompt: raw prompt, no system prompt.
Parser: rule-based constraint checker from IFEval official eval.
Score: strict (rule must pass) + loose (rule passes after light normalization) accuracy.
Loader: lukaemon/bbh · 23 sub-tasks · 6,500 Qs.
Prompt: per-sub-task template (causal judgement, web of lies, dyck languages, multistep arithmetic, etc.).
Parser: regex for the canonical answer format per task.
Score: macro-avg across 23 sub-tasks + per-sub-task breakdown.
Loader: lmsys/mt_bench_human_judgments · 80 multi-turn prompts × 8 categories.
Protocol: generate both turns → submit to GPT-4-as-judge (Anthropic API, claude-3.5-sonnet fallback).
Parser: judge returns 1–10 per turn, final score = avg of (turn1 + turn2)/2.
Score: overall mean + per-category mean + 95% bootstrap CI.
Loader: tatsu-lab/alpaca_eval · 805 Qs.
Protocol: generate → submit to GPT-4-Turbo-1106 as judge, win-rate vs GPT-4 baseline (length-controlled).
Score: LC win-rate (length-controlled) + raw win-rate.
Loader: live human preference votes · LMSYS collects blind A/B pairs from real users.
Protocol: route SOV33_small / SOV33_large into the Arena pool under a de-anonymized alias only after ≥ 5,000 votes are cast; category breakdown after 10K votes.
Score: Elo + per-category Elo + 95% CI. Public leaderboard published at lmarena.ai.
sov33-models/sov33-core (model change, prompt change, or wrapper change) → automatic eval kickoffhermes harness run sov33_large mmlu_pro --n=12032/sov33-runs/{run_id}/
├─ config.json # model + bench + git_sha + python_ver + torch_ver
├─ row_chain.jsonl # one signed row per question
├─ summary.json # aggregate metrics
├─ root_receipt.json # SIGIL root + verify key + BFT quorum + care_score
└─ reproduce.sh # one-command replay
| Benchmark | Pass threshold | Reason |
|---|---|---|
| MMLU-Pro | ≥ 70% | beats published Qwen3-30B-A3B baseline |
| GSM8K | ≥ 94% | approaches frontier on grade-school math |
| AIME 2024 | ≥ 23/30 | o1-class performance tier |
| IFEval | ≥ 80% strict | reliable instruction-following |
| BBH | ≥ 70% macro | hard reasoning parity |
| MT-Bench | ≥ 8.5/10 | GPT-4-judge parity |
| AlpacaEval | ≥ 50% LC win-rate | beats GPT-4 on length-controlled |
| Arena Elo | ≥ 1180 | top-3% of submitted models |