All external HTTP/HTTPS traffic entering the cluster is protected by CrowdSec, a collaborative, behaviour-based intrusion detection and prevention system. CrowdSec runs as native Kubernetes workloads and integrates directly with the Istio ingress gateway via the Envoy ext_authz gRPC protocol.
| Component | Detail |
|---|---|
| Chart | crowdsecurity/crowdsec v0.22.1 (appVersion v1.7.6) |
| Namespace | crowdsec |
| Sync Wave | 3 |
| LAPI | 1× Deployment (crowdsec-lapi), NFS RWX PVCs (1 Gi data, 100 Mi config) |
| Agents | 4× DaemonSet pods (one per node: control-plane + node1/2/3) |
| AppSec | 1× Deployment (crowdsec-appsec), port 7422, OWASP CRS + virtual-patching |
| Bouncer | crowdsec-envoy-bouncer Deployment in crowdsec ns — Envoy ext_authz gRPC protocol |
| Integration | Istio EnvoyFilter on homelab-gateway intercepts all requests, forwards to bouncer |
| CAPI | Connected — community blocklist pulling enabled |
Every HTTPS request hitting the Istio gateway passes through the crowdsec-ext-authz EnvoyFilter before reaching any backend. The EnvoyFilter calls the Envoy bouncer (crowdsec-envoy-bouncer.crowdsec.svc.cluster.local:8080) via gRPC. The bouncer checks the source IP against LAPI's in-memory ban cache (refreshed every 30 seconds). Banned IPs receive a 403 Forbidden immediately — no backend service is hit.
Fail-open: failure_mode_allow: true — if the bouncer pod is unreachable, requests flow through. This prevents a CrowdSec outage from taking down the entire cluster.
Each allowed request is also forwarded by the bouncer to the AppSec component which uses a custom appsec-config (custom/crs-and-vpatch) loading all rules as inband (real-time blocking):
| Rule Set | Loaded as | Coverage |
|---|---|---|
crowdsecurity/base-config |
Inband | ModSecurity base configuration, anomaly score variables |
crowdsecurity/vpatch-* |
Inband | ~130 specific CVE virtual patches (.env, .git/config, Spring4Shell, etc.) |
crowdsecurity/crs |
Inband | OWASP CRS with full anomaly scoring: SQLi, XSS, RCE, LFI, RFI, Log4Shell, Shellshock, etc. |
CRS anomaly scoring: Individual CRS rules add to tx.inbound_anomaly_score. REQUEST-949-BLOCKING-EVALUATION blocks the request when the total score ≥ threshold (default: 5). Multi-pattern SQLi/XSS payloads accumulate score and trigger a single deny.
CRS rule exclusion: Rule 911100 (HTTP Method Enforcement — blocks PUT/PATCH/DELETE) is disabled via RemoveInBandRuleByID(911100) on load. Standard REST methods are legitimate; blocking them adds no real security and breaks Vaultwarden and other REST APIs.
LAN bypass: Traffic from
clientTrustedIPs(LAN CIDRs + pod CIDR) completely skips the bouncer check. LAN addresses are also whitelisted at the agent level viacrowdsecurity/whitelist-good-actors.
Agents run as a DaemonSet on all 4 nodes and read logs from two sources:
| Source | Acquisition method | Detects |
|---|---|---|
| Istio IngressGateway access logs | Kubernetes API (podName: istio-ingressgateway-*) — nginx-format access log configured in istiod meshConfig |
HTTP attacks, CVE probes, bad user agents |
/var/log/auth.log |
hostPath file (hostVarLog: true) |
SSH brute force |
When an IP triggers enough suspicious events within a time window, the agent pushes an alert to the LAPI. The LAPI applies the default profile (ban for 4 h) and propagates the decision to the bouncer within ≤30 s.
The nginx parser (crowdsecurity/nginx collection) is used to parse Istio's access logs because Istio's IngressGateway is configured to emit nginx-compatible JSON access logs.
The LAPI is connected to api.crowdsec.net. Known malicious IPs from the global CrowdSec community (~400 k instances worldwide) are blocked automatically before they attempt an attack.
| Collection | Key Scenarios |
|---|---|
crowdsecurity/nginx |
Nginx JSON log parsing — the base for all HTTP detection (parses Istio IngressGateway access logs) |
crowdsecurity/base-http-scenarios |
Generic brute force, path probing, bad user agents, aggressive crawling, backdoor path requests, admin interface probing (/wp-admin, /phpMyAdmin), sensitive files (.env, .git), path traversal, SQL injection patterns, XSS patterns |
crowdsecurity/http-cve |
18 named CVEs: Log4Shell (CVE-2021-44228), Spring4Shell (CVE-2022-22965), Grafana path traversal (CVE-2021-43798), Confluence RCE, Apache traversal, VMware vCenter, Fortinet, F5 BIG-IP, PHP unit RCE, Text4Shell |
| Scenario | Description |
|---|---|
ssh-bf |
Classic SSH brute force (many failed logins) |
ssh-slow-bf |
Distributed / slow SSH brute force |
ssh-time-based-bf |
Time-pattern based SSH attacks |
ssh-cve-2024-6387 |
RegreSSHion CVE exploit attempts |
| Rule Set | Inband Config | Test Result |
|---|---|---|
crowdsecurity/base-config |
custom/crs-and-vpatch |
✅ Base config loaded (ModSec variables initialized) |
crowdsecurity/vpatch-* |
custom/crs-and-vpatch |
✅ .env, .git/config → HTTP 403 |
crowdsecurity/crs |
custom/crs-and-vpatch |
✅ SQLi, XSS, Log4Shell, Shellshock, RCE → HTTP 403 |
Every ban decision triggers a ntfy notification to the homelab-security topic:
kv/crowdsec/ntfy, injected as $NTFY_TOKEN env var — never in Githomelab-securitydefault_ip_remediation — all ban decisions trigger the notification# values.yaml — config.notifications.http.yaml
format: |
🚫 CrowdSec Ban{{ range . }}
IP: {{.Source.Value}}
Scenario: {{.Scenario}}
Duration: {{(index .Decisions 0).Duration}}{{ end }}
url: "https://ntfy.vyanh.uk/homelab-security"
method: POST
headers:
Authorization: "Bearer ${NTFY_TOKEN}"
Title: "CrowdSec Ban"
Priority: "high"
Tags: "no_entry"
Common mistake: The
models.Alerttype does NOT have.Valueor.Durationfields at the top level. Use.Source.Valuefor the IP and(index .Decisions 0).Durationfor the ban duration.
Traffic from these CIDRs skips the bouncer check (Istio AuthorizationPolicy + LAPI whitelist):
| CIDR | Network |
|---|---|
192.168.88.0/24 |
Main LAN (work VLAN) |
192.168.20.0/24 |
Home VLAN |
192.168.100.0/24 |
WireGuard VPN |
10.244.0.0/16 |
K8s pod CIDR |
Additionally, crowdsecurity/whitelist-good-actors whitelists all RFC1918 ranges at the agent level — LAN traffic passing the bouncer whitelist is also whitelisted by the detection pipeline, preventing false positives from internal hosts.
LAN-only services are also protected by an Istio AuthorizationPolicy (local-only in istio-system) that DENYs access from non-LAN source IPs to sensitive internal hostnames (vault, minio, proxmox, gitlab, portainer, etc.).
| Gap | Detail |
|---|---|
| East-west traffic | Pod-to-pod traffic inside the cluster never hits the gateway — lateral movement is not detected. Requires Cilium network policies or mTLS enforcement. |
| Layer 4 attacks | Raw TCP floods and port scans at the network level. CrowdSec is HTTP-focused. |
| Container runtime | Privilege escalation, container escape, cryptomining. Falco (wave 4) complements this. |
| K8s API server | kubectl / API-level attacks. Audit logging is not configured on this cluster. |
| NAS services | Portainer, Immich, Paperless etc. run outside K8s — their logs are not acquired. |
| Stream latency | ≤30 s window between ban decision and enforcement. |
| GitLab WAF bypass | Git smart HTTP (POST /git-upload-pack with packed binary objects) triggers OWASP CRS anomaly score 5 — false positive. GitLab is LAN-only so WAF is disabled for gitlab.homelab.vyanh.uk via a separate crowdsec-ext-authz-gitlab-disable EnvoyFilter. |
| Vault Path | K8s Secret | Namespace | Purpose |
|---|---|---|---|
kv/crowdsec/bouncer |
crowdsec-bouncer-key |
crowdsec |
LAPI pre-registers the Envoy bouncer with this API key |
kv/crowdsec/ntfy |
crowdsec-ntfy |
crowdsec |
ntfy token for ban notifications to homelab-security topic |
Previous Traefik dual-path pattern: Before migrating to Istio, the bouncer key also lived at
kv/traefik/crowdsec-bouncerin thetraefiknamespace. This is no longer needed — the Envoy bouncer runs inside thecrowdsecnamespace and reads the key directly.
| Item | Location |
|---|---|
| ArgoCD Application | k8s-cluster-config/core-components/crowdsec/application.yaml |
| Helm values | k8s-cluster-config/core-components/crowdsec/values.yaml |
| VSO resources | k8s-cluster-config/core-components/crowdsec/resources/ |
| EnvoyFilter (ext_authz) | k8s-cluster-config/core-components/istio-gateway/resources/crowdsec-authz.yaml |
| LAN-only AuthorizationPolicy | k8s-cluster-config/core-components/istio-gateway/resources/crowdsec-authz.yaml |
| LAPI service | crowdsec-service.crowdsec.svc.cluster.local:8080 |
| AppSec service | crowdsec-appsec-service.crowdsec.svc.cluster.local:7422 |
# List all active ban decisions
kubectl exec -n crowdsec deploy/crowdsec-lapi -- cscli decisions list
# Manually ban an IP (4 hour default)
kubectl exec -n crowdsec deploy/crowdsec-lapi -- cscli decisions add --ip 1.2.3.4
# Manually unban an IP
kubectl exec -n crowdsec deploy/crowdsec-lapi -- cscli decisions delete --ip 1.2.3.4
# Check registered machines (agents + appsec)
kubectl exec -n crowdsec deploy/crowdsec-lapi -- cscli machines list
# Check registered bouncers
kubectl exec -n crowdsec deploy/crowdsec-lapi -- cscli bouncers list
# Check recent alerts (last 10)
kubectl exec -n crowdsec deploy/crowdsec-lapi -- cscli alerts list --limit 10
# Check collections on a specific agent
kubectl exec -n crowdsec <agent-pod> -- cscli collections list
# Check AppSec collections and metrics
kubectl exec -n crowdsec deploy/crowdsec-appsec -- cscli collections list
kubectl exec -n crowdsec deploy/crowdsec-appsec -- cscli metrics
# Check acquisition pipeline metrics (parsed/whitelisted counts per source)
kubectl exec -n crowdsec <agent-pod> -- cscli metrics show acquisition
# Check CAPI status
kubectl exec -n crowdsec deploy/crowdsec-lapi -- cscli console status
# Force hard refresh in ArgoCD (clears manifest cache)
kubectl annotate application -n argocd crowdsec argocd.argoproj.io/refresh=hard --overwrite
# Re-register agents if LAPI was replaced (triggers fresh init containers)
kubectl rollout restart ds/crowdsec-agent -n crowdsec
kubectl rollout restart deploy/crowdsec-appsec -n crowdsec
Incident (2026-03-06): LAPI was configured with Longhorn RWO PVCs. When the LAPI pod rescheduled to a different node, Longhorn threw a Multi-Attach error. The new pod started with an emptyDir — corrupting the SQLite WAL. Machine registrations from all 4 agents and AppSec were lost. Agents fell into CrashLoopBackOff with "machine not found" errors for 12+ hours.
Fix: Switch LAPI PVCs to nfs-synology StorageClass with ReadWriteMany. NFS has no single-node attachment constraint.
# values.yaml — correct PVC config
lapi:
persistentVolume:
data:
storageClassName: nfs-synology
accessModes: [ReadWriteMany]
size: 1Gi
config:
storageClassName: nfs-synology
accessModes: [ReadWriteMany]
size: 100Mi
The wait-for-lapi-and-register init container runs cscli lapi register exactly once per pod lifetime. If the LAPI database is lost, agents will fail with "machine not found" on every restart.
Recovery: Force new agent pods (which re-run the init container):
kubectl rollout restart ds/crowdsec-agent -n crowdsec
kubectl rollout restart deploy/crowdsec-appsec -n crowdsec
externalTrafficPolicy: Local Required for Real Client IPsWith externalTrafficPolicy: Cluster (the default), kube-proxy SNATs external traffic to pod network IPs. The Istio gateway logs this internal IP as the source. crowdsecurity/whitelist-good-actors then whitelists all 10.0.0.0/8 — every HTTP log line is dropped by the detection pipeline.
Fix: externalTrafficPolicy: Local is set on the Istio gateway LoadBalancer service via the Gateway resource annotation. MetalLB L2 mode announces the VIP only from the node running the gateway pod, so traffic arrives without SNAT, and Istio logs the real client IP.
crowdsecurity/virtual-patching Does NOT Load CRSUsing appsec_config: crowdsecurity/virtual-patching means CRS is never loaded — SQLi, XSS, Log4Shell return HTTP 200. Use a custom appsec-config that explicitly lists crowdsecurity/crs as an inband rule:
appsec:
configs:
crs-and-vpatch.yaml: |
name: custom/crs-and-vpatch
inband_rules:
- crowdsecurity/base-config
- crowdsecurity/vpatch-*
- crowdsecurity/crs
https://crowdsec.github.io/helm-charts → 404https://crowdsecurity.github.io/helm-charts (org is crowdsecurity, not crowdsec)source in additionalAcquisitionThe chart JSON schema requires a source field in every additionalAcquisition entry:
additionalAcquisition:
- source: file # required by chart schema
filenames:
- /var/log/auth.log
labels:
type: syslog
crowdsec NamespaceSymptom (2026-04-27): Envoy bouncer logs operation not permitted / auth timeouts when checking decisions with LAPI. CrowdSec metrics show crowdsec_http_responseCode_total never incrementing for bouncer calls.
Root cause: The LAPI CiliumNetworkPolicy ingress rule for the bouncer used io.kubernetes.pod.namespace: istio-system — matching where Istio gateway pods run, not where the bouncer pod runs. The bouncer is deployed in the crowdsec namespace.
Fix:
# core-components/crowdsec/resources/netpol.yaml — LAPI CNP ingress
- fromEndpoints:
- matchLabels:
app: crowdsec-envoy-bouncer
io.kubernetes.pod.namespace: crowdsec # bouncer is in crowdsec ns
toPorts:
- ports:
- port: "8080"
protocol: TCP
After applying the CNP fix, restart the bouncer deployment to clear stale gRPC connections:
kubectl rollout restart deployment/crowdsec-envoy-bouncer -n crowdsec
Symptom: CI pipeline or other tools get 403 Forbidden on PATCH/PUT requests with binary bodies. The 403 comes from the ext_authz bouncer before reaching the backend.
Root cause: OWASP CRS anomaly scoring flags binary request bodies with scores of 5–10, exceeding the blocking threshold.
Diagnosis:
kubectl -n crowdsec logs deploy/crowdsec-appsec --since=30m | grep "WAF block"
Fix: Add the relevant source CIDR to clientTrustedIPs, or disable the EnvoyFilter for the specific hostname using a per-route ExtAuthzPerRoute override (as done for GitLab).