A comprehensive guide for operators of the DEFONEOS Sovereign AI OS. This knowledge base provides diagnostic steps, common issue resolutions, and best practices for maintaining system health and performance.
Regular health checks are crucial for proactive issue detection. These commands provide a baseline for system status.
Ensure all core DEFONEOS services are running and accessible.
systemctl status defoneos-core-api defoneos-bft-agent defoneos-data-pipeline
curl -sf http://localhost:8080/health || echo "Core API DOWN"
curl -sf http://localhost:8081/health || echo "BFT Agent DOWN"Review recent logs for errors or warnings across all components.
journalctl -u defoneos-core-api -n 50 --since "5 minutes ago"
journalctl -u defoneos-bft-agent -n 50 --since "5 minutes ago"
grep -E "ERROR|WARN" /var/log/defoneos/*.log | tail -20Addressing typical problems encountered during initial deployment or updates.
Verify that all environment variables and configuration files are correctly set.
cat /etc/defoneos/config.yaml
printenv | grep DEFONEOS_Resolution: Cross-reference with deployment documentation. Restart services after config changes.
Ensure necessary ports are open for inter-service communication and external access.
sudo ufw status verbose
sudo firewall-cmd --list-allResolution: Open ports 8080 (Core API), 8081 (BFT Agent), and any MCP-specific ports.
Diagnosing and resolving issues with Micro-Contextual Processor (MCP) server communication.
Confirm the MCP server is running and its network endpoint is accessible.
ping <MCP_IP_OR_HOSTNAME>
curl -sf http://<MCP_IP_OR_HOSTNAME>:<MCP_PORT>/health || echo "MCP <MCP_NAME> DOWN"Resolution: Check MCP server logs. Verify network routes and firewall rules.
Ensure API keys or tokens are correctly configured and have appropriate permissions.
grep -i "auth_token" /etc/defoneos/mcp-creds.yaml
# Example for a specific MCP
python3 -c "import mcporter; print(mcporter.auth_test('my_mcp_name'))"Resolution: Regenerate or update credentials. Verify scope and expiry.
Troubleshooting issues related to data flow into the DEFONEOS knowledge base and processing pipelines.
Check the status of the data ingestion queue and processing workers.
docker compose ps
docker compose logs data-ingestor | tail -50Resolution: Restart ingestion workers. Check database connectivity and disk space.
Identify and rectify issues with incoming data formats that prevent processing.
# Example: Check a data source for common parsing errors
python3 /opt/defoneos/scripts/validate_data_source.py --source <SOURCE_ID> --sample-size 10Resolution: Review data source schemas. Implement pre-processing sanitization steps.
Resolving inconsistencies or delays in the BFT Council's distributed state synchronization.
Monitor the BFT agent's log for signs of network partitioning or slow peer responses.
journalctl -u defoneos-bft-agent -n 100 | grep "consensus_lag_detected"
# Check peer connectivity
python3 -c "import bft_council_cli; print(bft_council_cli.peer_status())"Resolution: Verify network connectivity between BFT nodes. Restart unresponsive peers.
First steps and immediate actions in case of a suspected security breach or anomaly.
Isolate affected systems, notify security teams, and preserve forensic evidence.
sudo systemctl stop defoneos-core-api defoneos-bft-agent
sudo rsync -av --ignore-errors /var/log/defoneos/ /mnt/forensics/logs_<TIMESTAMP>/
# Initiate incident response playbook
/opt/defoneos/scripts/ir_playbook.sh --stage initial_containmentResolution: Follow the DEFONEOS Incident Response Policy. Engage designated security personnel.