DEFONEOS Post-Market Monitoring Plan

EU AI Act Article 17 · NIST AI RMF Monitor Function · ISO/IEC 23894
Version 1.0 · 5 Jul 2026 · Ed25519: c3f6a9d2e1b4f705 · Annex IV Section 4(g) compliant

Under EU AI Act Article 17, providers of high-risk AI systems must establish and document a post-market monitoring system in a proportionate manner. DEFONEOS implements this as a fully autonomous, continuous monitoring pipeline that tracks AI system performance, detects drift, correlates incidents, and generates regulatory reports — all SIGIL-audited.

5min

Evaluation Cycle

Monitoring Vectors

280

Checks Per Cycle

72h

Regulatory Report SLA

1. Article 17 Requirements → DEFONEOS Implementation

Art 17 Requirement	How DEFONEOS Satisfies	Evidence
17(1): System established & documented	This document + Gods-Eye CISO scanner configuration + 14 monitoring vectors	SIGIL: P\|system\|pmm-plan-v1.0\|...
17(1): Proportionate to risk	Monitoring intensity scaled by AI system risk classification (minimal/limited/high/unacceptable)	Risk classifier MCP
17(1): Agreed with national authority	PENDING — Monitoring design document ready for authority review	This page + System Card
17(2): Monitoring plan (Annex IV 4g)	This document constitutes the post-market monitoring plan as required by Annex IV	Full text below
Art 72: Serious incident reporting	SEV-1 incident → 72h regulatory notification pipeline (ICO + AI authority)	Incident Response Playbook
Art 73: Reporting to authorities	Automated report generation from SIGIL chain → regulatory submission (human approval required)	DORADO training export

2. The 14 Monitoring Vectors

V1: Model Performance Drift

What: Tracks output quality degradation over time using held-out evaluation sets.

Threshold: Performance drop > 5% from baseline → SEV-2 alert

Frequency: Every evaluation cycle (5 min)

V2: Input Distribution Shift

What: Detects when live input data diverges from training distribution (KL divergence, Population Stability Index).

Threshold: PSI > 0.25 → SEV-2 alert (significant shift)

V3: Bias & Fairness Metrics

What: Monitors demographic parity, equalised odds, disparate impact across protected characteristics.

Threshold: Disparate impact ratio < 0.8 → SEV-1 alert (4/5 rule)

V4: Adversarial Robustness

What: Probes with adversarial inputs (FGSM, PGD, prompt injection variants).

Threshold: Any adversarial success → SEV-2 alert

V5: Red-Line Compliance

What: Checks all 7 immutable red lines on every agent action.

Threshold: Any violation → SEV-1 CRITICAL (immediate halt)

V6: SIGIL Chain Integrity

What: Verifies hash chain continuity, signature validity, Bitcoin anchor (when available).

Threshold: Any hash mismatch → SEV-1 (tampering suspected)

V7: MCP Fleet Health

What: Polls all 30 MCP servers for availability, response time, error rate.

Threshold: Any MCP down > 60s → SEV-2. Error rate > 5% → SEV-3.

V8: BFT Council Liveness

What: Verifies 33-agent council quorum availability, vote patterns, consensus health.

Threshold: Fewer than 23 agents reachable → SEV-1 (governance failure)

V9: Foreign Access Detection

What: DORADO foreign IP detector + bot scanner. Non-sovereign access attempts logged.

Threshold: Any confirmed foreign access → SEV-1. Suspicious pattern → SEV-2.

V10: Data Sovereignty

What: Verifies no data has crossed UK borders. Network flow analysis.

Threshold: Any egress to non-UK IP → SEV-1 (sovereignty breach)

V11: Human Oversight Channel

What: Verifies human override channels are operational. Override latency measured.

Threshold: Override channel down → SEV-1 (safety system failure)

V12: Key & Certificate Health

What: Ed25519 key rotation schedule, certificate expiry, PQC migration readiness.

Threshold: Key within 30 days of rotation → SEV-3. Expired key → SEV-1.

V13: Resource Utilisation

What: CPU, memory, disk, network. Detects DoS patterns or resource exhaustion.

Threshold: CPU > 90% sustained 5min → SEV-3. Disk > 85% → SEV-2.

V14: Serious Incident Correlator

What: Cross-references all monitoring vectors. If multiple vectors trigger simultaneously, auto-escalates severity.

Threshold: 3+ vectors in SEV-2+ → auto-escalate to SEV-1. Correlation analysis feeds Art 73 reporting.

3. Monitoring Data Pipeline

┌─────────────────────────────────────────────────────────────┐
│              POST-MARKET MONITORING PIPELINE                │
├─────────────────────────────────────────────────────────────┤
│                                                              │
│  ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐               │
│  │ 14     │ │ 280    │ │ Drift  │ │ Bias   │               │
│  │Vectors │→│ Checks │→│ Detect │→│ Audit  │               │
│  └────────┘ └────────┘ └────────┘ └────────┘               │
│       │                                    │                  │
│       │           ┌────────┐              │                  │
│       └──────────▶│CORREL- │◀─────────────┘                  │
│                   │ATOR    │                                 │
│                   └────┬───┘                                 │
│                        │                                     │
│              ┌─────────┼─────────┐                           │
│              ▼         ▼         ▼                           │
│        ┌─────────┐┌─────────┐┌─────────┐                    │
│        │SIGIL    ││BFT      ││Regulatory│                    │
│        │Log     ││Council  ││Report   │                    │
│        └─────────┘└─────────┘└─────────┘                    │
│                        │                      │               │
│                        ▼                      ▼               │
│                 ┌──────────┐          ┌──────────┐           │
│                 │ Response │          │ Art 73   │           │
│                 │ Playbook │          │ Submit   │           │
│                 └──────────┘          └──────────┘           │
│                                           (Human gate)        │
└─────────────────────────────────────────────────────────────┘

4. Drift Detection Methodology

Method	Use Case	Threshold	DEFONEOS Application
Population Stability Index (PSI)	Input distribution shift	PSI > 0.25 = significant	Sensor input streams vs baseline
KL Divergence	Output distribution change	KL > 0.1 = warning	Agent decision distribution vs training
Page-Hinkley Test	Sudden change detection	Threshold = 50	Performance metric sudden drops
ADWIN (Adaptive Window)	Concept drift streaming	δ = 0.002	Real-time stream monitoring
Confidence Calibration	Model over/underconfidence	ECE > 0.1 = miscalibrated	Agent confidence vs accuracy

5. Regulatory Reporting Workflow

Step	Action	Timeline	Authority
1	Serious incident detected and classified SEV-1	T+0	Autonomous
2	BFT council validates incident + confirms severity	T+30s	BFT 23/33
3	Containment actions executed	T+2min	BFT + System
4	Incident report auto-drafted from SIGIL chain	T+1h	Autonomous
5	DPO reviews and approves report	T+24h	DPO (human)
6	Report submitted to relevant authority (ICO / AI authority)	T+72h max	DPO + Nick (human)
7	Follow-up actions tracked in OSCAL POA&M	T+7d	Autonomous
8	Corrective actions implemented and verified	T+30d	BFT + DPO

6. Monitoring Report Schema (Art 17)

{
  "report_type": "post_market_monitoring",
  "report_period": {
    "start": "2026-07-01T00:00:00Z",
    "end": "2026-07-05T16:00:00Z"
  },
  "system_identification": {
    "system_name": "DEFONEOS",
    "version": "1.0",
    "risk_classification": "high_risk",
    "conformity_assessment": "pending"
  },
  "monitoring_summary": {
    "total_cycles": 1440,
    "total_checks": 403200,
    "vectors_monitored": 14,
    "alerts_generated": {
      "sev1": 0,
      "sev2": 2,
      "sev3": 7,
      "sev4": 23
    }
  },
  "drift_analysis": {
    "input_shift_psi": 0.08,
    "output_shift_kl": 0.03,
    "performance_delta": -1.2,
    "calibration_ece": 0.04,
    "verdict": "STABLE — within acceptable thresholds"
  },
  "bias_audit": {
    "demographic_parity": 0.92,
    "disparate_impact": 0.88,
    "equalised_odds": 0.91,
    "verdict": "PASS — above 0.8 threshold"
  },
  "incidents": [],
  "corrective_actions": [],
  "sigil_attestation": {
    "chain_height": 847291,
    "chain_hash": "a3f7c2d8...",
    "signature": "ed25519:7f3a2b8c..."
  },
  "prepared_by": "DEFONEOS autonomous monitoring",
  "approved_by": null,
  "approval_required": true
}

7. Corrective Action Framework

Trigger	Action Class	Implementation	Verification
Drift PSI > 0.25	Model retraining	OLM router retrained on recent data	PSI returns to < 0.1
Bias DI < 0.8	Bias correction	Re-weighting, augmentation, threshold adjustment	DI returns to > 0.8
Adversarial vulnerability	Adversarial training	Adversarial examples added to training set	Red-team re-test passes
Performance degradation	Model rollback	Rollback to last-known-good model version	Performance restored to baseline
Red-line violation	System halt + review	Full halt, BFT review, charter amendment	Human authority sign-off

8. Stakeholder Reporting Cadence

Stakeholder	Frequency	Format	Content
Internal (CSOAI team)	Real-time	Dashboards	All 14 vectors, all alerts
Deploying org (e.g., MOD)	Daily digest	Automated email	Summary stats, SEV-2+ incidents
DPO	Real-time (SEV-1), Weekly	SIGIL + Email	All incidents, bias audit, drift
Regulators (ICO/AI Authority)	Per incident (72h), Quarterly	Formal report	Serious incidents, corrective actions, PMM summary
Public transparency	Monthly	SIGIL public explorer	Aggregated, anonymised metrics

📋 Honesty Register

"Continuous 5-min monitoring": The Gods-Eye CISO scanner runs at this cadence by design. Actual production deployment should verify the cycle holds under load.
Drift detection methods: PSI, KL divergence, Page-Hinkley, ADWIN, and calibration metrics are standard ML monitoring approaches. DEFONEOS implements them as designed — not yet benchmarked in production.
Bias metrics: The metrics (demographic parity, disparate impact, equalised odds) require ground-truth labels by protected characteristic. In a defence context, this may not always be available or appropriate.
72h regulatory reporting: The pipeline generates the report autonomously. Actual submission to regulators is a human gate (DPO + Nick).
"High-risk" classification: DEFONEOS is designed to be compatible with EU AI Act high-risk requirements. Formal conformity assessment has not been conducted.
Quarterly regulator reports: Designed and planned. No regulator relationship exists yet.

9. Alignment with International Frameworks

Framework	Article/Control	DEFONEOS Coverage
EU AI Act	Art 17 — Post-Market Monitoring	Full plan (this document)
EU AI Act	Art 72 — Serious Incident Reporting	72h pipeline implemented
EU AI Act	Art 73 — Reporting to Market Surveillance	Report auto-generation + human submission
EU AI Act	Annex IV 4(g) — Monitoring Plan	This document fulfils the requirement
NIST AI RMF 1.0	MEASURE 2.3-2.7 + MONITOR	Continuous evaluation, drift, bias, robustness
ISO/IEC 23894	Clause 7 — Monitoring	Risk monitoring, incident management integration
ISO/IEC 42001	Clause 9 — Performance Evaluation	Monitoring, measurement, analysis, evaluation
UK AI Safety Institute	Pre-deployment + Post-deployment	System Card + this PMM plan