All external HTTP/HTTPS traffic entering the cluster is protected by CrowdSec, a collaborative, behaviour-based intrusion detection and prevention system. CrowdSec runs as native Kubernetes workloads and integrates directly with the Istio ingress gateway via the Envoy ext_authz gRPC protocol.
| Component | Detail |
|---|---|
| Chart | crowdsecurity/crowdsec v0.24.0 |
| Namespace | crowdsec |
| Sync Wave | 3 |
| LAPI | 1× Deployment (crowdsec-lapi), NFS RWX PVCs (1 Gi data, 100 Mi config) |
| Agents | DaemonSet — one pod per node (control-plane + workers), container name crowdsec-agent; LAPI container is crowdsec-lapi |
| AppSec | 1× Deployment (crowdsec-appsec), port 7422, OWASP CRS + virtual-patching |
| Bouncer | crowdsec-envoy-bouncer Deployment in crowdsec ns — Envoy ext_authz gRPC protocol |
| Integration | Istio EnvoyFilter on homelab-gateway intercepts all requests, forwards to bouncer |
| CAPI | Connected — community blocklist pulling enabled |
Every HTTPS request hitting the Istio gateway passes through the crowdsec-ext-authz EnvoyFilter before reaching any backend. The EnvoyFilter calls the Envoy bouncer (crowdsec-envoy-bouncer.crowdsec.svc.cluster.local:8080) via gRPC. The bouncer checks the source IP against LAPI's in-memory ban cache (refreshed every 30 seconds). Banned IPs receive a 403 Forbidden immediately — no backend service is hit.
Fail-open: failure_mode_allow: true — if the bouncer pod is unreachable, requests flow through. This prevents a CrowdSec outage from taking down the entire cluster.
Each allowed request is also forwarded by the bouncer to the AppSec component which uses a custom appsec-config (custom/crs-and-vpatch) loading all rules as inband (real-time blocking):
| Rule Set | Loaded as | Coverage |
|---|---|---|
crowdsecurity/base-config |
Inband | ModSecurity base configuration, anomaly score variables |
crowdsecurity/vpatch-* |
Inband | ~130 specific CVE virtual patches (.env, .git/config, Spring4Shell, etc.) |
crowdsecurity/crs |
Inband | OWASP CRS with full anomaly scoring: SQLi, XSS, RCE, LFI, RFI, Log4Shell, Shellshock, etc. |
CRS anomaly scoring: Individual CRS rules add to tx.inbound_anomaly_score. REQUEST-949-BLOCKING-EVALUATION blocks the request when the total score ≥ threshold (default: 5). Multi-pattern SQLi/XSS payloads accumulate score and trigger a single deny.
CRS rule exclusion: Rule 911100 (HTTP Method Enforcement — blocks PUT/PATCH/DELETE) is disabled via RemoveInBandRuleByID(911100) on load. Standard REST methods are legitimate; blocking them adds no real security and breaks Vaultwarden and other REST APIs.
LAN bypass: Traffic from
clientTrustedIPs(LAN CIDRs + pod CIDR) completely skips the bouncer check. LAN addresses are also whitelisted at the agent level viacrowdsecurity/whitelist-good-actors.
Agents run as a DaemonSet on every node and read logs from two sources:
| Source | Acquisition method | Detects |
|---|---|---|
| Gateway API gateway access logs | Tails /var/log/containers/homelab-gateway-istio-*_istio-system_*.log (chart agent.acquisition[].podName: homelab-gateway-istio-*) — nginx-format access log configured in istiod meshConfig. poll_without_inotify: true (see gotcha below). |
HTTP attacks, CVE probes, bad user agents |
/var/log/auth.log |
hostPath file (hostVarLog: true) |
SSH brute force |
The pod glob is
homelab-gateway-istio-*(Gateway API deployment), not the oldistio-ingressgateway-*— that stale glob matched nothing after the Traefik→Istio migration and left HTTP detection silently dead (fixed 2026-07-03). Because it's a DaemonSet tailing node-local files, only the nodes currently hosting a gateway pod read gateway logs; the others legitimately show 0 lines for that source.
When an IP triggers enough suspicious events within a time window, the agent pushes an alert to the LAPI. The LAPI applies the default profile (ban for 4 h) and propagates the decision to the bouncer within ≤30 s.
The nginx parser (crowdsecurity/nginx collection) is used to parse Istio's access logs because Istio's IngressGateway is configured to emit nginx-compatible JSON access logs.
The LAPI is connected to api.crowdsec.net. Known malicious IPs from the global CrowdSec community (~400 k instances worldwide) are blocked automatically before they attempt an attack.
| Collection | Key Scenarios |
|---|---|
crowdsecurity/nginx |
Nginx JSON log parsing — the base for all HTTP detection (parses Istio IngressGateway access logs) |
crowdsecurity/base-http-scenarios |
Generic brute force, path probing, bad user agents, aggressive crawling, backdoor path requests, admin interface probing (/wp-admin, /phpMyAdmin), sensitive files (.env, .git), path traversal, SQL injection patterns, XSS patterns |
crowdsecurity/http-cve |
18 named CVEs: Log4Shell (CVE-2021-44228), Spring4Shell (CVE-2022-22965), Grafana path traversal (CVE-2021-43798), Confluence RCE, Apache traversal, VMware vCenter, Fortinet, F5 BIG-IP, PHP unit RCE, Text4Shell |
| Scenario | Description |
|---|---|
ssh-bf |
Classic SSH brute force (many failed logins) |
ssh-slow-bf |
Distributed / slow SSH brute force |
ssh-time-based-bf |
Time-pattern based SSH attacks |
ssh-cve-2024-6387 |
RegreSSHion CVE exploit attempts |
| Rule Set | Inband Config | Test Result |
|---|---|---|
crowdsecurity/base-config |
custom/crs-and-vpatch |
✅ Base config loaded (ModSec variables initialized) |
crowdsecurity/vpatch-* |
custom/crs-and-vpatch |
✅ .env, .git/config → HTTP 403 |
crowdsecurity/crs |
custom/crs-and-vpatch |
✅ SQLi, XSS, Log4Shell, Shellshock, RCE → HTTP 403 |
Ban decisions notify the homelab-security topic, tiered by severity since 2026-07-05 (before that every ban pushed instantly at high priority — ~85 pushes/day of scanner noise, since each attacker IP trips 4–6 http-* scenarios):
| Profile (match order) | Scenarios | Plugin | Delivery |
|---|---|---|---|
portscan_ip_remediation |
*portscan* |
ntfy-batch |
30 min batch / flush at 50 |
appsec_ip_remediation |
*appsec* (WAF blocks) |
ntfy |
instant, high priority |
cve_ip_remediation |
*CVE-* (exploit attempts) |
ntfy |
instant, high priority |
http_aggression_remediation |
http-flood / probing / crawl | ntfy-digest |
6 h digest / flush at 30 |
default_ip_remediation |
everything else | ntfy-digest |
6 h digest / flush at 30 |
kv/crowdsec/ntfy, injected as $NTFY_TOKEN env var — never in Githomelab-securitygroup_wait + group_threshold on the type: http notification plugin; profiles/plugins are read at LAPI startup, so a config change needs kubectl rollout restart deploy/crowdsec-lapi -n crowdsec after ArgoCD sync.CrowdSecAlertStorm (security group) fires a single warning when scenario overflows exceed 150/h for 15 min — baseline peak is ~70/h on a bad day. Triage with sort_desc(sum by (name) (increase(cs_bucket_overflowed_total[1h]))).# values.yaml — config.notifications.http.yaml
format: |
🚫 CrowdSec Ban{{ range . }}
IP: {{.Source.Value}}
Scenario: {{.Scenario}}
Duration: {{(index .Decisions 0).Duration}}{{ end }}
url: "https://ntfy.vyanh.uk/homelab-security"
method: POST
headers:
Authorization: "Bearer ${NTFY_TOKEN}"
Title: "CrowdSec Ban"
Priority: "high"
Tags: "no_entry"
Common mistake: The
models.Alerttype does NOT have.Valueor.Durationfields at the top level. Use.Source.Valuefor the IP and(index .Decisions 0).Durationfor the ban duration.
Traffic from these CIDRs skips the bouncer check (Istio AuthorizationPolicy + LAPI whitelist):
| CIDR | Network |
|---|---|
192.168.88.0/24 |
Main LAN (work VLAN) |
192.168.20.0/24 |
Home VLAN |
192.168.100.0/24 |
WireGuard VPN |
10.244.0.0/16 |
K8s pod CIDR |
Additionally, crowdsecurity/whitelist-good-actors whitelists all RFC1918 ranges at the agent level — LAN traffic passing the bouncer whitelist is also whitelisted by the detection pipeline, preventing false positives from internal hosts.
LAN-only services are also protected by an Istio AuthorizationPolicy (local-only in istio-system) that DENYs access from non-LAN source IPs to sensitive internal hostnames (vault, minio, proxmox, gitlab, portainer, etc.).
| Gap | Detail |
|---|---|
| East-west traffic | Pod-to-pod traffic inside the cluster never hits the gateway — lateral movement is not detected. Requires Cilium network policies or mTLS enforcement. |
| Layer 4 attacks | Raw TCP floods and port scans at the network level. CrowdSec is HTTP-focused. |
| Container runtime | Privilege escalation, container escape, cryptomining. Falco (wave 4) complements this. |
| K8s API server | kubectl / API-level attacks. Audit logging is not configured on this cluster. |
| NAS services | Portainer, Immich, Paperless etc. run outside K8s — their logs are not acquired. |
| Stream latency | ≤30 s window between ban decision and enforcement. |
| GitLab WAF bypass | Git smart HTTP (POST /git-upload-pack with packed binary objects) triggers OWASP CRS anomaly score 5 — false positive. GitLab is LAN-only so WAF is disabled for gitlab.homelab.vyanh.uk via a separate crowdsec-ext-authz-gitlab-disable EnvoyFilter. |
Full-stack scan (edge, cluster posture, certs, NAS, git secrets). Clean baseline: no anonymous cluster access, Immich/Technitium admin not internet-exposed, public apps enforce API auth, PSA baseline already blocks privileged in several namespaces, certs healthy.
Fixed (GitOps):
disallow-privileged-containers flipped Audit → Enforce. Excluded infra namespaces: kube-system, kyverno, rook-ceph, istio-system, cilium, falco, crowdsec, suricata, metallb-system, nas-ingress. block-latest-tag / require-resource-limits stay Audit (125 / 227 existing violations — enforcing would break syncs).sidecar.istio.io/inject:"false" — the istio-proxy wasn't exiting on Job completion, leaving pods at 0/1 that looked like failed backups (they were succeeding).Open (needs a maintenance window / dashboard access):
Dragonfab161093@ — committed in plaintext in immich-standby.yaml (immich-db Secret) and frigate/resources/configmap.yaml (camera RTSP), and reused as the Proxmox andy@pam / MikroTik / pve-exporter password. Rotate everywhere, move to Vault/VSO, then purge from git history. Lock-out risk → do supervised.utility/cloudflare_dns_update/update_cloudfare.sh; move to Vault.privileged: true (both NAS primary + k8s standby). Dropping it needs a fuse-device-plugin DaemonSet — k8s has no device-cgroup allowlist, so SYS_ADMIN + /dev/fuse alone gives fusermount3: Operation not permitted.Suricata (wave 7, suricata namespace, host-network pod on talos-cp3) complements CrowdSec with signature-based packet inspection. False-positive/noise suppressions live in threshold.conf inside k8s-cluster-config/core-components/suricata/resources/configmap-suricata.yaml — that file is the canonical list (25+ entries); each entry carries a comment explaining the why. After editing: commit → ArgoCD sync suricata → kubectl rollout restart deploy/suricata -n suricata (threshold.conf is only read at startup).
Suppression policy: scope as narrowly as possible — single host or pod CIDR as track by_src, never blanket-disable a signature that could still indicate exfil/intrusion from other sources. Prefer track by_dst instead when the false positive is really about what's being talked to (an internal service) rather than who's talking — a by_src node/pod-range suppression also blinds every other destination that source ever egresses to, including a real leak from a different pod/process sharing that same source IP (e.g. Cilium node-SNAT'd pod egress).
Notable host-scoped suppressions:
| Sig ID | Rule | Scope | Why |
|---|---|---|---|
| 2054140 | ET INFO External IP Lookup (checkip.amazonaws.com) | 10.244.0.0/16 |
vault-ui pod does one lookup at startup |
| 2054140 | ET INFO External IP Lookup (checkip.amazonaws.com) | 192.168.88.94 (Andy's iPhone) |
NordVPN iOS app polls checkip via NordVPN DNS every ~15 min, 3 queries per check — flooded homelab-security (2026-07-12). Other LAN hosts still alert (exfil indicator kept). iPhone lease made static on MikroTik so the IP can't drift away from the suppression. |
| 2003068 | ET SCAN Potential SSH Scan OUTBOUND | 192.168.88.103 (VM100/clawdbot) |
frequent git-over-SSH to github.com trips the rule (2026-07-06) |
| 2012888 | ET INFO Http Client Body contains pwd= in cleartext | by_dst 192.168.88.19 (NAS) |
Talos node -> NAS ntfy :2586 push flooded homelab-security hard enough to 429 the topic itself (2026-07-23, talos-worker6 -> .19:2586). Scoped by_dst on the NAS host, not by_src on the node range — the node IP is also the SNAT source for other pod egress, so a by_src rule would've hidden a real pwd= leak from any other pod on that node to any other destination. Any LAN device sending real pwd= creds to a host other than the NAS still alerts. |
The instant, unbatched ntfy_suricata sink in configmap-vector.yaml (Vector sidecar, high_severity = sev 1/2 alerts only) has no rate limit by default — unlike CrowdSec's tiered ntfy profiles, one alert in = one push out. A single noisy/new signature can flood homelab-security into a 429 storm before anyone's awake to write a threshold.conf suppression (happened with sig 2012888, 2026-07-23).
Fix: a throttle transform sits between high_severity and format_ntfy, keyed on signature_id + src_ip, capping each pair to 3 pushes per 15 min. VictoriaLogs still receives every raw event unthrottled (input is parse_eve, upstream of the throttle) — only the push-notification path is capped, so nothing is lost for later triage.
# configmap-vector.yaml — transforms
add_throttle_key:
type: remap
inputs: [high_severity]
source: |
.throttle_key = (to_string(.alert.signature_id) ?? "?") + "_" + (to_string(.src_ip) ?? "?")
throttle_alerts:
type: throttle
inputs: [add_throttle_key]
threshold: 3
window_secs: 900
key_field: .throttle_key
format_ntfy:
inputs: [throttle_alerts] # was [high_severity]
...
This is a backstop, not a replacement for suppressions — a signature that's genuinely noisy should still get a proper threshold.conf entry; the throttle just buys time so it doesn't page anyone in the meantime.
| Vault Path | K8s Secret | Namespace | Purpose |
|---|---|---|---|
kv/crowdsec/bouncer |
crowdsec-bouncer-key |
crowdsec |
LAPI pre-registers the Envoy bouncer with this API key |
kv/crowdsec/ntfy |
crowdsec-ntfy |
crowdsec |
ntfy token for ban notifications to homelab-security topic |
Previous Traefik dual-path pattern: Before migrating to Istio, the bouncer key also lived at
kv/traefik/crowdsec-bouncerin thetraefiknamespace. This is no longer needed — the Envoy bouncer runs inside thecrowdsecnamespace and reads the key directly.
| Item | Location |
|---|---|
| ArgoCD Application | k8s-cluster-config/core-components/crowdsec/application.yaml |
| Helm values | k8s-cluster-config/core-components/crowdsec/values.yaml |
| VSO resources | k8s-cluster-config/core-components/crowdsec/resources/ |
| EnvoyFilter (ext_authz) | k8s-cluster-config/core-components/istio-gateway/resources/crowdsec-authz.yaml |
| LAN-only AuthorizationPolicy | k8s-cluster-config/core-components/istio-gateway/resources/crowdsec-authz.yaml |
| LAPI service | crowdsec-service.crowdsec.svc.cluster.local:8080 |
| AppSec service | crowdsec-appsec-service.crowdsec.svc.cluster.local:7422 |
# List all active ban decisions
kubectl exec -n crowdsec deploy/crowdsec-lapi -- cscli decisions list
# Manually ban an IP (4 hour default)
kubectl exec -n crowdsec deploy/crowdsec-lapi -- cscli decisions add --ip 1.2.3.4
# Manually unban an IP
kubectl exec -n crowdsec deploy/crowdsec-lapi -- cscli decisions delete --ip 1.2.3.4
# Check registered machines (agents + appsec)
kubectl exec -n crowdsec deploy/crowdsec-lapi -- cscli machines list
# Check registered bouncers
kubectl exec -n crowdsec deploy/crowdsec-lapi -- cscli bouncers list
# Check recent alerts (last 10)
kubectl exec -n crowdsec deploy/crowdsec-lapi -- cscli alerts list --limit 10
# Check collections on a specific agent
kubectl exec -n crowdsec <agent-pod> -- cscli collections list
# Check AppSec collections and metrics
kubectl exec -n crowdsec deploy/crowdsec-appsec -- cscli collections list
kubectl exec -n crowdsec deploy/crowdsec-appsec -- cscli metrics
# Check acquisition pipeline metrics (parsed/whitelisted counts per source)
kubectl exec -n crowdsec <agent-pod> -- cscli metrics show acquisition
# Check CAPI status
kubectl exec -n crowdsec deploy/crowdsec-lapi -- cscli console status
# Force hard refresh in ArgoCD (clears manifest cache)
kubectl annotate application -n argocd crowdsec argocd.argoproj.io/refresh=hard --overwrite
# Re-register agents if LAPI was replaced (triggers fresh init containers)
kubectl rollout restart ds/crowdsec-agent -n crowdsec
kubectl rollout restart deploy/crowdsec-appsec -n crowdsec
Incident (2026-03-06): LAPI was configured with Longhorn RWO PVCs. When the LAPI pod rescheduled to a different node, Longhorn threw a Multi-Attach error. The new pod started with an emptyDir — corrupting the SQLite WAL. Machine registrations from all 4 agents and AppSec were lost. Agents fell into CrashLoopBackOff with "machine not found" errors for 12+ hours.
Fix: Switch LAPI PVCs to nfs-synology StorageClass with ReadWriteMany. NFS has no single-node attachment constraint.
# values.yaml — correct PVC config
lapi:
persistentVolume:
data:
storageClassName: nfs-synology
accessModes: [ReadWriteMany]
size: 1Gi
config:
storageClassName: nfs-synology
accessModes: [ReadWriteMany]
size: 100Mi
The wait-for-lapi-and-register init container runs cscli lapi register exactly once per pod lifetime. If the LAPI database is lost, agents will fail with "machine not found" on every restart.
Recovery: Force new agent pods (which re-run the init container):
kubectl rollout restart ds/crowdsec-agent -n crowdsec
kubectl rollout restart deploy/crowdsec-appsec -n crowdsec
externalTrafficPolicy: Local Required for Real Client IPsWith externalTrafficPolicy: Cluster (the default), kube-proxy SNATs external traffic to pod network IPs. The Istio gateway logs this internal IP as the source. crowdsecurity/whitelist-good-actors then whitelists all 10.0.0.0/8 — every HTTP log line is dropped by the detection pipeline.
Fix: externalTrafficPolicy: Local is set on the Istio gateway LoadBalancer service via the Gateway resource annotation. MetalLB L2 mode announces the VIP only from the node running the gateway pod, so traffic arrives without SNAT, and Istio logs the real client IP.
crowdsecurity/virtual-patching Does NOT Load CRSUsing appsec_config: crowdsecurity/virtual-patching means CRS is never loaded — SQLi, XSS, Log4Shell return HTTP 200. Use a custom appsec-config that explicitly lists crowdsecurity/crs as an inband rule:
appsec:
configs:
crs-and-vpatch.yaml: |
name: custom/crs-and-vpatch
inband_rules:
- crowdsecurity/base-config
- crowdsecurity/vpatch-*
- crowdsecurity/crs
poll_without_inotify: trueSymptom (2026-07-09): CrowdsecNoGatewayLogs fires — agents read gateway logs fine for a while, then cs_filesource_hits_total freezes (rate(...[15m]) == 0) even though the gateway is actively serving and the log file's mtime is current. cscli metrics show acquisition shows a stale, non-advancing line count.
Root cause: the chart hardcodes force_inotify: true on the generated acquis file, and /var/log/containers/*.log are symlinks that containerd repoints on rotation (containerLogMaxSize, ~10 Mi). Over the hostPath/overlayfs mount, inotify misses the rotation and the tailer stays stuck on the dead inode — reading nothing further.
Fix: set poll_without_inotify: true on the acquisition item so the tailer polls (re-stats the path) and follows the new file after each rotation:
# core-components/crowdsec/values.yaml
agent:
acquisition:
- namespace: istio-system
podName: homelab-gateway-istio-*
program: nginx
poll_without_inotify: true # else tail freezes on container-log rotation
The chart's configmap-checksum annotation auto-rolls the agent DaemonSet on sync — no manual restart needed. Verify the counter advances, not merely that it is > 0: cscli metrics show acquisition twice a few seconds apart, or sum(rate(cs_filesource_hits_total{source=~".*homelab-gateway.*"}[5m])) > 0.
https://crowdsec.github.io/helm-charts → 404https://crowdsecurity.github.io/helm-charts (org is crowdsecurity, not crowdsec)source in additionalAcquisitionThe chart JSON schema requires a source field in every additionalAcquisition entry:
additionalAcquisition:
- source: file # required by chart schema
filenames:
- /var/log/auth.log
labels:
type: syslog
crowdsec NamespaceSymptom (2026-04-27): Envoy bouncer logs operation not permitted / auth timeouts when checking decisions with LAPI. CrowdSec metrics show crowdsec_http_responseCode_total never incrementing for bouncer calls.
Root cause: The LAPI CiliumNetworkPolicy ingress rule for the bouncer used io.kubernetes.pod.namespace: istio-system — matching where Istio gateway pods run, not where the bouncer pod runs. The bouncer is deployed in the crowdsec namespace.
Fix:
# core-components/crowdsec/resources/netpol.yaml — LAPI CNP ingress
- fromEndpoints:
- matchLabels:
app: crowdsec-envoy-bouncer
io.kubernetes.pod.namespace: crowdsec # bouncer is in crowdsec ns
toPorts:
- ports:
- port: "8080"
protocol: TCP
After applying the CNP fix, restart the bouncer deployment to clear stale gRPC connections:
kubectl rollout restart deployment/crowdsec-envoy-bouncer -n crowdsec
Symptom: CI pipeline or other tools get 403 Forbidden on PATCH/PUT requests with binary bodies. The 403 comes from the ext_authz bouncer before reaching the backend.
Root cause: OWASP CRS anomaly scoring flags binary request bodies with scores of 5–10, exceeding the blocking threshold.
Diagnosis:
kubectl -n crowdsec logs deploy/crowdsec-appsec --since=30m | grep "WAF block"
Fix: Add the relevant source CIDR to clientTrustedIPs, or disable the EnvoyFilter for the specific hostname using a per-route ExtAuthzPerRoute override (as done for GitLab).