This page documents significant outages with root cause analysis, resolution steps, and preventive measures.
Date: 2026-02 | Severity: High | Duration: ~30 minutes
K8s workloads using TrueNAS NFS PVCs became completely unresponsive due to NFS server thread exhaustion. TrueNAS's KVM process pinned at 100% CPU and the VM became unreachable via SSH and web UI.
TrueNAS SCALE defaults to 2 NFS server threads (servers: 2). This is sufficient for a home NAS with a few SMB clients, but dangerously low for Kubernetes:
The cluster had approximately 15+ pods with NFS PVCs at the time, each capable of concurrent I/O.
| Affected | Not Affected |
|---|---|
All pods with nfs-subdir-retain PVCs |
Pods using longhorn storage |
All pods with nfs-subdir-delete PVCs |
Pods with no persistent storage |
Grafana (was on nfs-subdir-retain) |
VictoriaMetrics (was also NFS at the time) |
| Harbor Registry, LifeOps DB | Vault, Authentik (Longhorn) |
Compounding factor: Monitoring (VictoriaMetrics) was also on TrueNAS NFS at the time. The outage took down monitoring during the outage — making it harder to diagnose what was failing.
# From Proxmox host — graceful reboot first
qm reboot 109
# If graceful times out after ~60s, force reset
qm reset 109
# After TrueNAS comes back, immediately increase NFS threads
curl -u "andy:<password>" -X PUT https://192.168.88.230/api/v2.0/nfs \
-H "Content-Type: application/json" \
-d '{"servers": 8}'
# Verify in TrueNAS web UI: Services > NFS > Edit > Servers = 8
K8s pods recover automatically once NFS is available again — no manual pod restarts needed (kubelet retries NFS mounts).
If TrueNAS NFS becomes unresponsive and SSH is closed:
# From Mac or any machine with Proxmox access
ssh [email protected] "qm reboot 109"
# Wait 60s, check if TrueNAS web UI at https://192.168.88.230 responds
# If not: qm reset 109
Date: 2026-02 | Severity: High | Duration: Several days (silent degradation)
K8s worker nodes silently lost RAM over several days due to Proxmox's memory balloon driver. This caused intermittent pod OOMKills and incorrect scheduling decisions. The degradation was invisible in both K8s and Proxmox dashboards — it looked like application bugs, not infrastructure.
Why kubelet doesn't notice: Kubelet reads allocatable memory at startup and caches it. The balloon driver shrinks the VM's physical memory without notifying the guest OS in a way that kubelet responds to. Kubelet continues advertising 12GB of allocatable memory to the scheduler even though the VM only has 4GB.
Why it's invisible in Proxmox: The Proxmox UI shows the VM's configured memory (12GB), not the current ballooned size. The balloon value only appears if you run pvesh get /nodes/andy/qemu/<vmid>/status/current.
Default Proxmox VM configuration sets balloon: 4096 (4GB minimum). When the Proxmox host experienced any memory pressure, the balloon driver silently reduced all 3 worker VMs from 12GB to 4GB over several hours.
The control plane (VMID 107) was not affected because it has fewer pods and less memory pressure — its balloon had not yet triggered.
# Check current balloon config on all VMs (run on Proxmox host)
pvesh get /nodes/andy/qemu --output-format=text | grep -E "name|mem |balloon"
# From K8s side — check what kubelet thinks is allocatable
kubectl describe node k8s-node1 | grep -A5 "Allocatable:"
# Compare to what the VM actually has
# Expected: ~11Gi allocatable on a 12GB VM (minus kernel/system)
# Actual during incident: ~3.5Gi (balloon shrank to 4GB)
# Disable ballooning on all K8s worker VMs (live — no VM restart required)
# Run on Proxmox host (192.168.88.100)
qm set 103 --balloon 0 # k8s-node2
qm set 104 --balloon 0 # k8s-node3
qm set 108 --balloon 0 # k8s-node1
# Restart kubelet on each worker to refresh allocatable resources
ssh [email protected] "qm guest exec 103 -- systemctl restart kubelet"
ssh [email protected] "qm guest exec 104 -- systemctl restart kubelet"
ssh [email protected] "qm guest exec 108 -- systemctl restart kubelet"
# Verify allocatable memory is now correct
kubectl describe node k8s-node1 | grep -A5 "Allocatable:"
# Should show ~11Gi
| VMID | Name | RAM | Balloon | Notes |
|---|---|---|---|---|
| 103 | k8s-node2 | 12GB | 0 (off) | Fixed |
| 104 | k8s-node3 | 12GB | 0 (off) | Fixed |
| 108 | k8s-node1 | 12GB | 0 (off) | Fixed |
| 107 | k8s-controlplane | 8GB | 2048 | Kept — fewer critical pods |
| 109 | TrueNAS-Scale | 16GB | 2048 | Kept — not K8s workload |
pvesh get on Proxmox host and kubectl describe node for allocatable vs configured memory mismatch.Date: 2026-02 | Severity: Medium | Duration: Until OTEL collector restored
LifeOps backend entered a crash loop (CrashLoopBackOff) due to a cascade: OTEL collector went down → backend goroutines blocked waiting for OTEL connection → /api/health responses slowed → liveness probe timeout → kubelet restart → repeat.
The LifeOps backend initialises an OTEL gRPC exporter at startup. When the OTEL collector endpoint is unreachable, the gRPC client enters a reconnection backoff loop. During this backoff, any goroutine that tries to export a span calls into the gRPC layer and blocks on a channel send waiting for the connection to become available.
In Go, if the OTEL span export call is synchronous (not fire-and-forget), the HTTP request handler goroutine blocks until the OTEL call either succeeds or times out. If there is no explicit timeout, it blocks indefinitely — or until the OTEL client eventually gives up (which may take 30+ seconds).
The liveness probe calls /api/health with a 1-second timeout. If the handler goroutine is blocked on OTEL, the probe times out and kubelet restarts the pod.
# Pod in crash loop
kubectl get pods -n life-ops
# lifeops-backend-xxx 0/1 CrashLoopBackOff 12 1h
# Logs show liveness probe failures
kubectl logs -n life-ops <pod> --previous
# context deadline exceeded (liveness probe timeout)
# OTEL collector is the root cause
kubectl get pods -n monitoring | grep otel
# otel-collector-xxx 0/1 ImagePullBackOff 0 2h
# Option 1: Fix the OTEL collector (preferred)
kubectl describe pod -n monitoring <otel-pod> # find the root cause
# Fix the image pull, OOM, config issue, etc.
# Option 2: Temporarily disable OTEL to stop the crash loop
kubectl set env deployment/lifeops-backend -n life-ops \
OTEL_EXPORTER_OTLP_ENDPOINT=""
# Once OTEL collector is healthy again, re-enable
kubectl set env deployment/lifeops-backend -n life-ops \
OTEL_EXPORTER_OTLP_ENDPOINT="http://otel-collector.monitoring:4317"
WithTimeout or use context cancellation in OTEL SDK config--previous pod logs AND look for unhealthy pods in other namespaces that the crashing pod depends onDate: 2026-03-06 → 2026-03-07 | Severity: High | Duration: ~12 h detection blindspot + silent Telegram failure since deployment
All 4 CrowdSec agents and the AppSec pod entered CrashLoopBackOff with "machine not found" errors after the LAPI pod was replaced. This was compounded by two pre-existing silent bugs discovered during investigation: Telegram notifications were never being sent due to a wrong template field name, and HTTP detection was completely blind due to a Traefik traffic policy misconfiguration.
The LAPI's SQLite database was stored in an emptyDir volume. emptyDir persists across container restarts within the same pod, but is destroyed when the pod itself is replaced.
When the LAPI pod was replaced (due to node reschedule after Longhorn RWO multi-attach error):
wait-for-lapi-and-register init containers do not re-run on container restarts — only on pod replacementCrashLoopBackOffThe agents were stuck until their pods were explicitly deleted (triggering new pods with fresh init containers).
Original Longhorn RWO issue: Before the emptyDir phase, the LAPI used a Longhorn RWO PVC. When the pod rescheduled to a different node, Longhorn's Multi-Attach error caused the new pod to start without the PVC (emptyDir fallback), corrupting the SQLite WAL.
The HTTP notifier template in values.yaml used {{.Value}} and {{.Duration}}:
# Wrong — causes silent template error
IP: {{.Value}}\nDuration: {{.Duration}}
# Correct — models.Alert fields
IP: {{.Source.Value}}\nDuration: {{(index .Decisions 0).Duration}}
The models.Alert type does not have .Value or .Duration at the top level. On every alert, the LAPI logged:
level=error msg="format alerts for notification: template: :1:69:
executing "" at <.Value>: can't evaluate field Value in type *models.Alert"
No Telegram messages were ever sent. This had been broken since the initial deployment.
externalTrafficPolicy: Cluster (Silent Since Deployment)With externalTrafficPolicy: Cluster, kube-proxy routes external traffic through any node and applies SNAT — rewriting the client source IP to the pod-network gateway (10.244.x.x). Traefik logs this internal IP as ClientHost.
The crowdsecurity/whitelist-good-actors parser whitelists 10.0.0.0/8. Every Traefik log line was silently whitelisted — 0 events ever reached any HTTP detection scenario.
Verified: 1130+ Traefik log lines processed, 1130 whitelisted, 0 poured to any bucket.
| Component | Status During Incident |
|---|---|
| Traefik bouncer (IP ban check) | ✅ Working (stream cache from last sync) |
| AppSec WAF (per-request block) | ✅ Working (blocks still applied) |
| HTTP scenario detection | ❌ Dead since deployment (all IPs whitelisted) |
| SSH brute force detection | ✅ Working (not affected by LAPI restart) |
| Telegram notifications | ❌ Dead since deployment (template bug) |
| Community blocklist (CAPI) | ✅ Working (pulled periodically) |
Step 1 — Immediate recovery (agent re-registration):
kubectl rollout restart ds/crowdsec-agent -n crowdsec
kubectl rollout restart deploy/crowdsec-appsec -n crowdsec
# All 6 pods Running, 0 restarts within ~2 minutes
Step 2 — Permanent fix (NFS PVCs):
# values.yaml — switched from Longhorn RWO to NFS RWX
lapi:
persistentVolume:
data:
enabled: true
storageClassName: nfs-synology
accessModes: [ReadWriteMany]
size: 1Gi
config:
enabled: true
storageClassName: nfs-synology
accessModes: [ReadWriteMany]
size: 100Mi
Verification: LAPI pod manually deleted → new pod came up → cscli machines list showed all agents still registered. No rollout restart needed.
Step 3 — Fix Telegram template:
# Before (broken):
format: '... IP: {{.Value}}\nDuration: {{.Duration}} ...'
# After (correct):
format: '... IP: {{.Source.Value}}\nDuration: {{(index .Decisions 0).Duration}} ...'
Step 4 — Fix Traefik source IP preservation:
# traefik values.yaml
service:
spec:
externalTrafficPolicy: Local # was Cluster
After fix: Traefik logs real client IPs. LAN traffic (192.168.88.x) is still RFC1918-whitelisted (correct). Internet attackers are now detected.
error level but there is no metric or alert. Consider monitoring LAPI error log rate.cscli metrics show acquisition to confirm lines are reaching detection buckets, not just being parsed and whitelisted.Date: 2026-03-15 | Severity: Medium | Duration: ~1h
backend and reminder-checker pods in life-ops stuck in ImagePullBackOfflifeops-backend, lifeops-frontend) had 0 artifactsThe Harbor retention policy for the applications project was configured with nDaysSinceLastPush: 7 (TTL-based, delete images older than 7 days). The last CI/CD run that built and pushed images was 2026-03-07 (8 days before). The daily midnight retention job on 2026-03-15 deleted all images since they were outside the 7-day window.
The Terraform module (modules/harbor/harbor.tf) had been updated to include a safety net Rule 2 (most_recently_pushed = 5 — always keep the 5 most recently pushed images regardless of age), but Terraform Cloud had not been re-applied since that change was written. Only Rule 1 existed in Harbor.
Additionally, the retention_days for applications in main.tf was already updated to 90 (correct), but again not applied.
Summary: two-layer protection existed in code but neither was in effect in Harbor.
applications: Rule 1 nDaysSinceLastPush: 90, Rule 2 latestPushedN: 5 ✓gha-apps: Rule 1 nDaysSinceLastPush: 7, Rule 2 latestPushedN: 5 ✓tooling-images: Rule 1 nDaysSinceLastPush: 90, Rule 2 latestPushedN: 5 ✓workflow_dispatch on AnhTran1610/LifeOps CI with component=all, skip_ci=true8d86bbaupdate-manifests job updated applications/lifeops/values.yaml in k8s-cluster-config1/1 Running)latestPushedN safety net.modules/harbor/harbor.tf are invisible to Harbor until Terraform Cloud actually runs. Verify retention policy rules in the Harbor UI after any Terraform module change.most_recently_pushed: 5 is the critical safety net — it ensures the currently-deployed image is never wiped regardless of push age.Date: 2026-03-14 | Severity: High | Duration: ~2h
CrashLoopBackOff — startup logs full of invalid record length at 0/...The OOM was the trigger; Longhorn's reaction amplified it into PG corruption.
256Mi) for the size of the manifest set being rendered.# 1. Scale Authentik Postgres StatefulSet to 0 — stop the crash loop
kubectl scale -n authentik statefulset authentik-postgresql --replicas=0
# 2. Spin up a sidecar pod with the same image + PVC mount, but no PG running
# (use a debug pod template that just sleeps)
# 3. Inside the sidecar:
pg_resetwal -f /bitnami/postgresql/data
# This discards the broken WAL segment and writes a fresh one.
# 4. Start Postgres again
kubectl scale -n authentik statefulset authentik-postgresql --replicas=1
# 5. Postgres comes up but indexes built during the crash window may be corrupt.
# Connect from the same sidecar and reindex everything:
psql -U postgres -c 'REINDEX DATABASE authentik;'
# 6. Restart Authentik server + worker so they reconnect cleanly
kubectl rollout restart -n authentik deploy/authentik-server deploy/authentik-worker
After the reindex Authentik server logged migrations applied, OIDC flows resumed.
256Mi to 1Gi in argocd-bootstrap (patch-repo-server-resources.yaml). Survives the manifest set we have today with ~3× headroom.pg_resetwal is the correct first move for "invalid record length" — but always follow with REINDEX, the same write that broke the WAL likely also broke an index.Date: 2026-04-10 | Severity: Critical | Duration: ~3h
pg_authid and pg_database files were zero-byteA pve3 NVMe-backed ZFS pool (Samsung 980 Pro) returned zero-filled reads for several blocks under the authentik-postgresql-0 Longhorn replica. ZFS scrub later flagged them as CKSUM errors. Two of the corrupted blocks happened to land in the Postgres system catalogs (pg_authid, pg_database) — making the database unrecoverable since Postgres cannot rebuild these from scratch.
The same NVMe event also damaged 23 other Longhorn volumes — recovery for those tracked separately as INC-009.
A clean re-init was the only viable path (no good backup to restore from at the time):
# 1. Scale StatefulSet to 0
kubectl scale -n authentik statefulset authentik-postgresql --replicas=0
# 2. Delete the PVC (StatefulSet is at 0 so no PV detach loop)
kubectl delete pvc -n authentik data-authentik-postgresql-0
# 3. Scale back up — StatefulSet recreates the PVC and Bitnami initContainer
# runs initdb fresh, picking up POSTGRES_PASSWORD from existingSecret
kubectl scale -n authentik statefulset authentik-postgresql --replicas=1
# 4. Authentik runs its own first-boot migrations on top of the empty DB
kubectl rollout restart -n authentik deploy/authentik-server deploy/authentik-worker
# 5. Re-create the akadmin user (Authentik bootstraps it on first boot)
kubectl exec -n authentik deploy/authentik-worker -- ak create_admin_group akadmin
# Set password to Dragonfab161093@ via the Authentik UI
# 6. ArgoCD re-applied the OIDC blueprints (ConfigMaps in resources/)
# automatically — each app got its provider + application back
All OIDC-secured apps recovered as soon as Authentik served /.well-known/openid-configuration again.
argocd, grafana, harbor blueprints). Authentik replays them on first boot, so the OIDC providers / applications self-configure with no manual click-through.authentik/<app>-oidc — VSO re-syncs them and the apps trust the new Authentik immediately.pg_authid — accept the reinit. Postgres has no --initdb-just-the-catalogs mode.zpool scrub — the silent zero-fill happened weeks before this was discovered. A weekly scrub would have surfaced the CKSUM errors much earlier.pg_dump cron in this namespace. Should mirror the wikijs / ente-auth pattern with daily MinIO uploads.Date: 2026-04-10 | Severity: Critical | Duration: ~6h end-to-end
Faulted after the Samsung NVMe event from INC-008Same NVMe-pool corruption on pve3 ZFS that caused INC-008. Longhorn engines on the affected node returned read errors → replicas marked Faulted → Longhorn began rebuilding from the surviving replicas onto healthy nodes.
Recovery was driven by Longhorn's built-in rebuild logic, supervised manually:
Volume → Replicas tab) — every volume had at least one good replica on a different node.zpool scrub on pve3 — confirmed CKSUM errors, replaced the bad NVMe, re-attached pve3.End state: all 23 volumes Healthy, 3 replicas each, one replica per node.
replicas: 3 is actually configured for new volumes — easy to forget on a one-off PVC and end up with 1-replica volumes that have no recovery path.Date: 2026-04-19 | Severity: Low (latent) | Duration: N/A — issue persists
When the internal CA was bootstrapped, the procedure planned to:
/tmp/vault-ca.keykv/infra/vault-internal-ca so it could be retrieved later for renewalsStep 3 was skipped. /tmp/vault-ca.key was then cleared (either by reboot or tmpwatch) before anyone noticed the kv path was empty. The leaf cert and its (still valid) public CA are stored in K8s Secrets and continue to work — but the private key needed to issue any new cert under the same CA is gone.
This is a latent failure: nothing is broken right now, but at cert renewal time we must rotate the CA itself.
vault-tls in the vault namespace) with the new leaf.VaultAuth resources reference a CA bundle ConfigMapvaultwarden-litefs and vault-watcher (NAS Docker) must be re-pointedkubectl exec scripts that talk to Vault directlyThis is non-trivial — CA rotation while keeping HA quorum is delicate.
Going forward, every new internal CA must be written to Vault before the source file is removed:
# Generate CA
openssl genrsa -out /tmp/ca.key 4096
# ...sign leaf cert...
# IMMEDIATELY stash the CA key in Vault before doing anything else
vault kv put kv/infra/<purpose>-ca \
ca_key="$(cat /tmp/ca.key)" \
ca_crt="$(cat /tmp/ca.crt)"
# THEN it is safe to remove /tmp files
shred -u /tmp/ca.key
/tmp is hostile to anything you might want later — any non-trivial private key should never live there even briefly.kv/infra/vault-internal-ca is empty at the end of the run, not silently leave a placeholder.Date: 2026-07-01 → 2026-07-02 | Severity: Medium | Duration: ~8h (overnight), 100 restarts per replica
CrashLoopBackOff (~100 restarts each); Nextcloud fully unavailableThe fix-php-conf-perms init container (added to work around runAsUser: 1024 not being able to write redis-session.ini into the container-internal /usr/local/etc/php/conf.d/) copies its own image's conf.d into an emptyDir that is then mounted over the main container's conf.d.
That init container was hard-pinned to nextcloud:32.0.5-apache in core-components/nextcloud/values.yaml, while Renovate chart bumps moved the main image to nextcloud:34.0.1-apache (PHP 8.5).
opcache.so and no docker-php-ext-opcache.ini in the new imageconf.d from the 32.0.5 image still contained docker-php-ext-opcache.ini (zend_extension=opcache)Warning: Failed loading Zend extension 'opcache' ... cannot open shared object filephp -r 'echo $OC_Version...' output to compare installed vs image version — the warning text polluted both version strings, breaking the sort -V comparisonCan't start Nextcloud because the version of the data (33.0.5.1) is higher than the docker image version (34.0.1.2) and downgrading is not supported → crashloopThe error message was a red herring: 33 → 34 is a perfectly valid single-major upgrade; the comparison itself was corrupted.
# ArgoCD: Synced + Degraded, only Deployment/nextcloud unhealthy
kubectl get pods -n nextcloud # both replicas 0/1 Running, ~100 restarts
kubectl logs -n nextcloud <pod> # interleaved Zend warnings INSIDE the version error text
The giveaway: the "Can't start Nextcloud" error message was interleaved with PHP warnings — the version numbers in the message were split across lines by the warning output, proving the captured version strings were polluted.
core-components/nextcloud/values.yaml from nextcloud:32.0.5-apache to nextcloud:34.0.1-apache (matching the chart's main image) — commit e9e49c5Upgrading nextcloud from 33.0.5.1 ... data upgrade (~2 min; second replica correctly waited on the init lock)occ status → version: 34.0.1.2, needsDbUpgrade: false; 2/2 ready; ArgoCD Healthyvalues.yaml images in this repo (only resources/ manifests), so this is a manual coupling.php stdout and any warning breaks it.Date: ~2026-05 → 2026-07-03 | Severity: Medium (silent security degradation) | Duration: ~2 months undetected
cscli decisions list empty — no local bans; combined with the bouncer's WAF enforcement being disabled (2026-05-27 fail-open outage), the edge enforced nothingistio-ingressgateway-*_istio-system_*.log; the real Gateway API pods are homelab-gateway-istio-*. The glob matched nothing → 0 lines read. (istio-gateway-* also exists but is idle — ~2 lines/hr vs ~460/10min on the real gateway.)%START_TIME% is ISO8601 (the nginx parser needs strftime), the client field printed the whole comma-separated XFF chain (parser needs one IP), and the YAML >- block scalar rendered \n as two literal characters after the user agent.community_pull=false: go-cs-bouncer v0.0.21 leaves DecisionsStreamOpts.CommunityPull at its false zero-value and kdwils/envoy-proxy-bouncer never sets it (no config knob). Upstream limitation, still open — fix requires an upstream PR or a self-built patched bouncer image (one line: b.Opts.CommunityPull = true).agent.acquisition[].podName: homelab-gateway-istio-*; removed the dead /var/log/auth.log source (Talos nodes have no auth.log)%REQ(X-ENVOY-EXTERNAL-ADDRESS?:DOWNSTREAM-REMOTE-ADDRESS-WITHOUT-PORT)% - -
[%START_TIME(%d/%b/%Y:%H:%M:%S %z)%] "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%"
%RESPONSE_CODE% %BYTES_SENT% "%REQ(REFERER)%" "%REQ(USER-AGENT)%"
> (folded-clip) for a single real trailing newline, plus defaultConfig.gatewayTopology.numTrustedProxies: 1 so Envoy resolves the real client IP behind cloudflared/SNAT (gateway pod restart required — ProxyConfig is not dynamic)cscli metrics show acquisition end-to-end (read > 0 AND parsed > 0 AND whitelisted ≈ LAN only). Pod green ≠ pipeline alive.community_pull=false there means the community blocklist is decorative.