🐉

Deployment Runbook

DEFONEOS — 90-day deployment on MOD Cloud or on-premise — v1.0

90 DAYS DOCKER

📋 Prerequisites

Requirement	Specification	Status
Cloud Account	MOD Cloud, AWS London, Azure UK South, or GCP London	☐ Provision
Compute (Minimum)	4 VMs: 8 vCPU, 32GB RAM each + 1 GPU (T4/A10G) for YOLOv8	☐ Provision
Storage	500GB SSD per VM + 2TB object storage for ISR data	☐ Provision
Network	VPC with public subnet (API) + private subnet (data) + NAT gateway	☐ Configure
DNS	defoneos.your-domain (A record → load balancer)	☐ Configure
TLS Certificate	Let's Encrypt or MOD PKI certificate	☐ Provision
Cyber Essentials	Valid certificate (required for MOD deployment)	HUMAN GATE
SC Clearance	For at least 1 team member (for classified work)	HUMAN GATE

🏗️ Architecture — What We're Deploying

┌─────────────────────────────────────────────────────┐
│                   LOAD BALANCER (443)                 │
│              (NGINX / AWS ALB / Azure App GW)          │
├──────────────┬──────────────┬──────────────┬──────────┤
│   API GW     │  Cesium COP  │  SIGIL Chain │  Admin   │
│   :8080      │  :3000       │  :9091       │  :4000   │
├──────────────┼──────────────┼──────────────┼──────────┤
│              │              │              │          │
│  ┌─────────┐ │ ┌──────────┐ │ ┌──────────┐ │ ┌──────┐ │
│  │ ISR     │ │ │ Fusion   │ │ │ SIGIL    │ │ │ Auth │ │
│  │ Engine  │ │ │ Engine   │ │ │ Ledger   │ │ │ OIDC │ │
│  │ :9090   │ │ │ :3001    │ │ │ :9091    │ │ │:4000 │ │
│  └────┬────┘ │ └────┬─────┘ │ └──────────┘ │ └──────┘ │
│       │      │      │                     │          │
│  ┌────┴────┐ │ ┌────┴─────┐         ┌─────┴────┐     │
│  │  30 MCP │ │ │ C2 FTS   │         │ BFT      │     │
│  │  Servers│ │ │ :8087    │         │ Council  │     │
│  │ :5001-  │ │ │          │         │ :9092    │     │
│  │ :5030   │ │ └──────────┘         └──────────┘     │
│  └─────────┘ │                                        │
│              │              ┌──────────────────────┐  │
│  ┌─────────┐ │              │  GPU NODE            │  │
│  │ Swarm   │ │              │  YOLOv8 :8500        │  │
│  │ Engine  │ │              │  (T4 / A10G)         │  │
│  │ :3002   │ │              └──────────────────────┘  │
│  └─────────┘ │                                        │
├──────────────┴──────────────┴────────────────────────┤
│              POSTGRESQL (SIGIL + Missions)             │
│              REDIS (Cache + Pub/Sub)                   │
│              MINIO / S3 (ISR Data Lake)               │
└───────────────────────────────────────────────────────┘

🚀 Phase 1: Infrastructure (Days 1-14)

Provision Cloud Resources

Create 4 VMs + 1 GPU instance via Terraform.

# Clone DEFONEOS infra repo
git clone https://github.com/csoai-org/defoneos-infra.git
cd defoneos-infra/terraform

# Configure variables
cp terraform.tfvars.example terraform.tfvars
# Edit terraform.tfvars: set cloud_provider, region, domain, certs

# Deploy
terraform init
terraform plan -out defoneos.plan
terraform apply defoneos.plan

# Output: 4 VM IPs + GPU IP + LB DNS

Configure Network Security

Set up VPC, security groups, mTLS certificates.

# Generate SPIFFE certificates for each service
./scripts/gen-spiffe-certs.sh

# Configure firewall rules
# Allow: 443 (LB), 8080-8090 (internal API), 9090-9092 (gRPC)
# Deny: All external except 443
# Internal: mTLS only (SPIFFE verified)

# Verify
./scripts/verify-network.sh

Deploy Database Layer

PostgreSQL for SIGIL chain + mission data. Redis for cache. MinIO for ISR data lake.

# Deploy via Docker Compose (Phase 1)
cd defoneos-infra/docker
docker compose -f docker-compose.db.yml up -d

# Verify
docker compose -f docker-compose.db.yml ps
# Expected: postgres (healthy), redis (healthy), minio (healthy)

# Initialize SIGIL chain
./scripts/init-sigil-chain.sh
# Output: SIGIL genesis entry created. Chain ID: defoneos-001

🔧 Phase 2: Core Services (Days 15-45)

Deploy API Gateway + Auth

cd defoneos-infra/docker

# Configure OAuth2 / OIDC
export OIDC_ISSUER=https://auth.your-domain
export OIDC_CLIENT_ID=defoneos-api
export OIDC_CLIENT_SECRET=

# Deploy API gateway
docker compose -f docker-compose.api.yml up -d

# Verify
curl -k https://localhost/api/v1/health
# Expected: {"status":"healthy","version":"1.0","sigil_chain":"valid"}

Deploy 30 MCP Sensor Servers

# Deploy all 30 MCPs via federation script
./scripts/deploy-mcp-fleet.sh

# Output:
# [1/30] sentinel-hub-mcp..... ✓ deployed on :5001
# [2/30] os-opendata-mcp...... ✓ deployed on :5002
# [3/30] data-gov-uk-mcp...... ✓ deployed on :5003
# ...
# [30/30] mqtt-bridge-mcp..... ✓ deployed on :5030
# All 30 MCPs online. Federation healthy.

# Verify each MCP
./scripts/verify-mcp-fleet.sh
# Expected: 30/30 ONLINE, 0/30 OFFLINE

Deploy Fusion Engine + Cesium COP

# Deploy fusion engine
docker compose -f docker-compose.fusion.yml up -d

# Deploy Cesium 3D globe frontend
docker compose -f docker-compose.cesium.yml up -d

# Verify fusion
curl -k https://localhost/api/v1/isr/fusion/cop?bbox=53.8,-1.5,53.9,-1.4
# Expected: JSON with satellite, maritime, air_quality, osint, iot layers

# Open COP in browser
echo "COP available at: https://cop.your-domain"

Deploy C2 Backbone (FreeTAKServer)

# Deploy FreeTAKServer
docker compose -f docker-compose.c2.yml up -d

# Configure CoT endpoints
./scripts/configure-c2.sh

# Verify C2
curl -k https://localhost/api/v1/c2/cot
# Expected: {"events":[],"status":"online","protocol":"CoT v1.0"}

Deploy BFT Council + SIGIL

# Deploy 33 BFT agents
./scripts/deploy-bft-council.sh

# Output:
# [1/33] Agent council-001....... ✓ online
# [2/33] Agent council-002....... ✓ online
# ...
# [33/33] Agent council-033...... ✓ online
# Quorum: 23/33. Council ready.

# Verify BFT
curl -k https://localhost/api/v1/govern/bft/status
# Expected: {"agents_online":33,"quorum":23,"pending_votes":0}

🤖 Phase 3: AI/ML Layer (Days 46-70)

Deploy YOLOv8 ISR Pipeline

# Deploy on GPU node
cd defoneos-infra/docker
docker compose -f docker-compose.gpu.yml up -d

# Load DEFONEOS-trained YOLOv8 model
docker exec defoneos-gpu python3 -c "
from ultralytics import YOLO
model = YOLO('models/yolov8-defoneos-v3.pt')
print(f'Model loaded: {model.info()}')
print(f'Classes: vehicle, person, structure, vessel, aircraft, animal')
"

# Verify inference
curl -k -X POST -F "image=@test_recon.jpg" \
  https://localhost/api/v1/isr/anomaly/detect
# Expected: {"detections":[...],"inference_ms":132}

Deploy Swarm Engine (Mava + PX4)

# Deploy PX4 SITL (simulation) first
docker compose -f docker-compose.swarm-sim.yml up -d

# Load Mava RL-trained policy
docker exec defoneos-swarm python3 -c "
from mava import SwarmPolicy
policy = SwarmPolicy.load('models/swarm-isr-v3.xml')
print(f'Policy loaded. Drones: {policy.num_drones}')
print(f'Scenario: {policy.scenario}')
"

# Verify simulation
curl -k -X POST -d '{"mission_type":"isr_recon","num_drones":4}' \
  https://localhost/api/v1/swarm/launch
# Expected: {"mission_id":"msn-test-001","status":"bft_pending"}

Deploy JSP 936 Compliance Generator

# Deploy compliance service
docker compose -f docker-compose.compliance.yml up -d

# Generate first compliance report
curl -k -X POST -d '{"model_id":"yolov8-defoneos-v3"}' \
  https://localhost/api/v1/govern/jsp936/report/generate

# Expected: {"report_id":"jsp936-001","model":"yolov8-defoneos-v3",
#   "status":"compliant","frameworks":["jsp936","uk_ai_bill","eu_ai_act","iso42001"]}

✅ Phase 4: Integration Testing & Demo (Days 71-90)

End-to-End System Test

# Run full E2E test suite
./scripts/e2e-test.sh

# Expected output:
# [PASS] MCP Federation: 30/30 servers responding
# [PASS] ISR Fusion: COP generation < 5 sec
# [PASS] YOLOv8: Detection latency < 150 ms
# [PASS] Swarm: 4-drone sim launch + RTL
# [PASS] C2: CoT publish/receive
# [PASS] BFT Council: Quorum reached in < 10 sec
# [PASS] SIGIL Chain: Integrity verified (15432 entries)
# [PASS] JSP 936: Report generated for all deployed models
# [PASS] Auth: OAuth2 token issuance + validation
# [PASS] Rate Limits: Throttling active
# Result: 10/10 PASS

Live Demo Scenario — Yorkshire Flood Response

# Trigger flood scenario
./scripts/demo-flood-scenario.sh

# This will:
# 1. Load Yorkshire Digital Twin on Cesium COP
# 2. Simulate flood water rise (River Aire, Leeds)
# 3. Launch 4-drone swarm for ISR coverage
# 4. Fuse satellite imagery + drone feeds + IoT sensors
# 5. Detect stranded vehicles/persons via YOLOv8
# 6. Publish CoT events to C2 for emergency services
# 7. Log everything to SIGIL chain
# 8. Generate JSP 936 compliance report
# Duration: ~8 minutes
# Output: Live COP + mission replay + audit trail

Generate Deployment Audit Package

# Generate full audit package
./scripts/gen-audit-package.sh

# Output: /opt/defoneos/audit-package/
#   ├── sigil-chain-export.jsonl    (Full SIGIL chain export)
#   ├── jsp936-reports/             (Compliance reports for all models)
#   ├── sbom/                       (Software Bill of Materials per container)
#   ├── pentest-report.pdf          (Security scan results)
#   ├── network-topology.pdf        (Architecture diagram)
#   └── deployment-manifest.yaml    (Full deployment config)

Handover to MOD Operator

# Create operator accounts
./scripts/create-operators.sh --role=c2_operator --count=3
./scripts/create-operators.sh --role=analyst --count=5
./scripts/create-operators.sh --role=swarm_commander --count=2
./scripts/create-operators.sh --role=auditor --count=2

# Print deployment summary
./scripts/deployment-summary.sh

# ==========================================
# DEFONEOS DEPLOYMENT COMPLETE
# ==========================================
# Platform:    https://cop.your-domain
# API:         https://api.your-domain
# SIGIL Chain: 15432+ entries (verified)
# MCP Servers: 30/30 ONLINE
# BFT Council: 33/33 ONLINE (quorum 23)
# YOLOv8:      mAP50 0.9234, 132ms inference
# Swarm:       Mava RL v3, PX4 SITL ready
# C2:          FreeTAKServer online (CoT)
# Compliance:  JSP 936 compliant
# Deploy time: 90 days
# ==========================================

🔄 Operational Maintenance

Task	Frequency	Automated
SIGIL chain backup	Daily (02:00)	✅ Cron
MCP health check	Every 5 min	✅ Auto
Container vulnerability scan	Daily (03:00)	✅ Trivy
Key rotation (Ed25519 + PQC)	Every 90 days	✅ Auto
BFT agent health	Every 1 min	✅ Auto
JSP 936 report refresh	Per model deploy	✅ Auto
Database backup	Daily (01:00)	✅ Auto
ISR data archival	Weekly	✅ Auto
Security patch review	Weekly	❌ Manual
MOD compliance review	Quarterly	❌ Manual

🚨 Troubleshooting

Symptom	Cause	Fix
MCP server offline	Upstream API down or rate limited	`./scripts/restart-mcp.sh sentinel-hub-mcp`
BFT quorum not reached	Too many agents offline (>10)	`./scripts/check-bft-agents.sh` + restart failed agents
YOLOv8 latency > 200ms	GPU memory contention	`nvidia-smi` → kill stale processes → restart inference server
Cesium COP blank	Fusion engine down or tile server offline	`docker logs defoneos-fusion` + check Redis
SIGIL chain invalid	Tampering or storage corruption	CRITICAL → halt all ops → restore from backup → investigate
Swarm won't launch	BFT veto or geofence misconfiguration	Check `/api/v1/govern/bft/status` + verify geofence polygon
Drone not responding	DTLS handshake failure or EM interference	Check CoT connectivity → verify pre-shared key → switch to MQTT fallback