HTKI Operations Report
AI Server Load Incident
Root cause identified
Incident report for the AI server load spike observed in Grafana on 2026-09-06.
Summary
The load spike was real and was not caused by Prometheus or Grafana rendering. The strongest matching cause is the Authentik LDAP outpost container ak-outpost-eventhacker-mail-ldap-outpost, which was repeatedly starting, panicking, logging a stack trace, and restarting.
Timeline
| Local time | 2026-09-06 12:15-12:18 CEST |
|---|---|
| Server time | 2026-09-06 10:15-10:18 UTC |
| Peak load | node_load1 = 18.15 at 2026-09-06 10:16:45 UTC |
| Peak CPU | Approximately 93% at 2026-09-06 10:16:30 UTC |
| Process pressure | node_procs_running = 32, node_procs_blocked = 0 |
Evidence
- Prometheus showed a real
node_load1peak of18.15. - CPU mode breakdown at peak was mostly user and system CPU, with practically no I/O wait.
- Systemd logged repeated starts and clean deactivations for Docker scope
docker-b672c0da6a98d0d333cd71c86cf0a7eaa2db56df0ec46c6ef7d2b74c80c5f8bc.scope. - The matching container was
ak-outpost-eventhacker-mail-ldap-outpost, imageghcr.io/goauthentik/ldap:2026.8.0. - Container logs repeatedly reported
no ldap provider defined, followed by a panic and shutdown. - The container restart policy was
unless-stopped, so Docker kept restarting the failed outpost. - The container had already accumulated a restart count of
14716during inspection.
Related Events
Cron jobs ran around 10:15 UTC, including the XvX AI agent metrics collector, a Codex agent status publisher, Nextcloud cron, and the Picologic cybercheck runner. These jobs were short-lived. The Picologic cybercheck log reported no due sites. Prometheus node-exporter apt and NVMe collectors ran shortly after the peak and finished normally.
Assessment
The incident was caused by process churn and logging from the broken Authentik LDAP outpost restart loop. It was not a disk wait issue and no Prometheus or node-exporter service fault was found in the inspected window.
Recommended Fix
- If the LDAP outpost is required, attach a valid LDAP provider in Authentik and redeploy the outpost.
- If the LDAP outpost is not required, stop and disable/remove the Docker container so it cannot continue restarting.
- Add an alert for high Docker restart counts on service containers.
- Reduce excessive debug logging for recurring probes or failed outposts once the service state is understood.