Secrets in the homelab are managed through HashiCorp Vault (3-pod HA Raft cluster) with the Vault Secrets Operator (VSO) automatically syncing secrets into Kubernetes.
| Setting | Value |
|---|---|
| Helm Chart | hashicorp/vault v0.31.0 |
| Namespace | vault |
| Sync Wave | 4 |
| URL | https://vault.homelab.vyanh.uk |
| Access | LAN-only (local-only ipAllowList middleware) |
| HA Mode | 3-pod Raft cluster |
| Unseal | Transit auto-unseal via vault-transit on NAS |
| KV Engine | v2, mounted at kv/, max 10 versions per secret |
| Audit Log | File at /vault/audit/vault-audit.log (auditStorage PVC) |
| Metrics | Prometheus via /v1/sys/metrics (unauthenticated, pod annotations) |
| Setting | Value |
|---|---|
| Helm Chart | hashicorp/vault-secrets-operator v1.0.1 |
| Namespace | vso-system |
| Sync Wave | 5 (after Vault) |
The following hardening measures are in place:
Enabled on all 3 Vault pods:
vault audit enable file file_path=/vault/audit/vault-audit.log
Backed by a dedicated 5Gi Longhorn PVC (auditStorage). All Vault API requests are logged.
vault-transit (NAS Docker) uses TLS with a self-signed certificate (10-year, IP SAN 192.168.88.19):
/volume1/docker/vault-transit/tls/tls.crtvault-transit-ca secret → /vault/userconfig/transit-tls/tls.crtaddress = "https://192.168.88.19:8201" with tls_ca_certRoot token is revoked after setup. Terraform uses a scoped orphan token:
terraform-k8s-infra (covers sys/auth, auth/, sys/mounts, sys/policies, kv/)vault_token and at kv/ops/terraform-cloudThe Vault IngressRoute has an ipAllowList middleware (local-only) restricting access to:
192.168.88.0/24, 192.168.20.0/24, 192.168.100.0/24, 10.244.0.0/16An init container (fix-raft-perms) runs before Vault on every pod start:
chmod 600 /vault/data/raft/raft.db
Prevents the rw-rw---- warning caused by Longhorn's default group permissions.
unauthenticated_metrics_access is set inside the listener "tcp" telemetry block (required for Vault 1.16+, not the top-level telemetry stanza):
listener "tcp" {
...
telemetry {
unauthenticated_metrics_access = true
}
}
VictoriaMetrics scrapes via pod annotations (prometheus.io/scrape: "true", scheme: https).
The transit unseal provider runs as a Docker container on the NAS:
| Setting | Value |
|---|---|
| Image | hashicorp/vault:1.18 |
| Port | 192.168.88.19:8201 (LAN-only) |
| TLS | Self-signed cert with IP SAN 192.168.88.19 |
| Storage | File at /volume1/docker/vault-transit/data/ |
| Token | Periodic 720h token stored in /vault/data/transit-token |
| Transit key | vault-unseal on transit/ mount |
The entrypoint script bootstraps Vault on first run (init, unseal, enable transit, create key, create token) and re-validates the token on every restart.
Updating the transit token in K8s after vault-transit restarts:
NEW_TOKEN=$(ssh nas 'docker exec vault-transit cat /vault/data/transit-token')
kubectl create secret generic vault-transit-token -n vault \
--from-literal=token="$NEW_TOKEN" --dry-run=client -o yaml | kubectl apply -f -
When root access is needed (e.g. to enable new features, revoke tokens):
# 1. Initialize generate-root — save the OTP from the output
kubectl exec -n vault vault-0 -- sh -c 'VAULT_SKIP_VERIFY=1 vault operator generate-root -init -format=json'
# 2. Submit 3 recovery keys (from vault-unseal-keys secret in vault namespace)
NONCE="<nonce from step 1>"
kubectl exec -n vault vault-0 -- sh -c "VAULT_SKIP_VERIFY=1 vault operator generate-root -nonce=$NONCE <key1>"
kubectl exec -n vault vault-0 -- sh -c "VAULT_SKIP_VERIFY=1 vault operator generate-root -nonce=$NONCE <key2>"
kubectl exec -n vault vault-0 -- sh -c "VAULT_SKIP_VERIFY=1 vault operator generate-root -nonce=$NONCE <key3>"
# 3. Decode the encoded token using the OTP
kubectl exec -n vault vault-0 -- sh -c 'VAULT_SKIP_VERIFY=1 vault operator generate-root -decode=<encoded_token> -otp=<otp> -format=json'
# 4. Use the token, then ALWAYS revoke it when done
VAULT_TOKEN=<root_token> vault token revoke -self
Recovery keys are in the vault-unseal-keys K8s secret in the vault namespace.
Important: Always create Terraform tokens as orphan (
-orphanflag). Non-orphan tokens are revoked when the parent (root) is revoked.
Every namespace that needs Vault secrets follows this exact three-file pattern in k8s-cluster-config/core-components/<app>/resources/:
sa.yaml)apiVersion: v1
kind: ServiceAccount
metadata:
name: vso-auth
namespace: <namespace>
vault-auth.yaml)apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
name: vso-auth
namespace: <namespace>
spec:
method: kubernetes
mount: kubernetes
kubernetes:
role: vso-<namespace>
serviceAccount: vso-auth
<name>-vaultstaticsecret.yaml)apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
name: <secret-name>
namespace: <namespace>
spec:
type: kv-v2
mount: kv
path: <namespace>/<secret-path>
refreshAfter: 1h # use 60s only for secrets that change frequently
vaultAuthRef: vso-auth
destination:
create: true
name: <k8s-secret-name>
refreshAfter tuning: Stable secrets (OIDC, S3/MinIO, TSIG) use
1h. Only use60sfor secrets that might be rotated frequently.
Some OIDC client secrets need to be accessible from two namespaces. This is solved by storing the same secret at two Vault paths:
| Application | Authentik Path | App Path | Purpose |
|---|---|---|---|
| ArgoCD | kv/authentik/argocd-oidc |
kv/argocd/oidc |
OIDC client secret |
| Grafana | kv/authentik/grafana-oidc |
kv/monitoring/grafana-oidc |
OIDC client secret |
| Harbor | kv/authentik/harbor-oidc |
kv/harbor/oidc |
OIDC client secret |
Both paths contain the same clientSecret value. The Authentik blueprint reads from kv/authentik/*, and the target app reads from its own namespace path.
Some VaultStaticSecrets use templates to transform Vault keys into different K8s Secret keys:
# Example: ArgoCD OIDC secret
spec:
destination:
create: true
name: argocd-oidc-secret
transformation:
templates:
oidc.authentik.clientSecret:
text: '{{ get .Secrets "clientSecret" }}'
This maps clientSecret from Vault to oidc.authentik.clientSecret in the K8s Secret, matching what ArgoCD expects.
| Vault Path | K8s Namespace | K8s Secret Name | Refresh |
|---|---|---|---|
kv/monitoring/grafana/admin |
monitoring | grafana-admin | 1h |
kv/monitoring/grafana-oidc |
monitoring | grafana-oidc-secret | 1h |
kv/monitoring/alertmanager/telegram |
monitoring | alertmanager-telegram | 60s |
kv/monitoring/alertmanager/ntfy |
monitoring | alertmanager-ntfy | 60s |
kv/monitoring/tempo-minio |
monitoring | tempo-minio | 1h |
kv/cloudflared-connector/tunnel |
cloudflared-connector | cloudflared-connector-secrets | 60s |
kv/authentik/secrets |
authentik | authentik-secrets | 60s |
kv/authentik/postgresql |
authentik | authentik-postgresql | 60s |
kv/authentik/argocd-oidc |
authentik | argocd-oidc-client-secret | 1h |
kv/authentik/grafana-oidc |
authentik | grafana-oidc-client-secret | 1h |
kv/authentik/harbor-oidc |
authentik | harbor-oidc-client-secret | 1h |
kv/authentik/db-backup-minio |
authentik | db-backup-minio | 1h |
kv/argocd/oidc |
argocd | argocd-oidc-secret | 1h |
kv/argocd/repos/lifeops |
argocd | lifeops-repo | 1h |
kv/harbor/oidc |
harbor | harbor-oidc-secret | 1h |
kv/harbor/registry-s3 |
harbor | harbor-registry-s3 | 1h |
kv/harbor/db-backup-minio |
harbor | db-backup-minio | 1h |
kv/nextcloud/admin |
nextcloud | nextcloud-admin | 60s |
kv/nextcloud/db-backup-minio |
nextcloud | db-backup-minio | 1h |
kv/technitium/admin |
technitium | technitium-admin | 60s |
kv/technitium/tsig |
technitium | technitium-tsig | 1h |
kv/external-dns/tsig |
external-dns | technitium-tsig | 1h |
kv/life-ops/secrets |
life-ops | lifeops-secrets | 1h |
kv/life-ops/db-backup-minio |
life-ops | db-backup-minio | 1h |
kv/crowdsec/bouncer |
crowdsec | crowdsec-bouncer-key | 1h |
kv/crowdsec/ntfy |
crowdsec | crowdsec-ntfy | 60s |
kv/traefik/crowdsec-bouncer |
traefik | crowdsec-bouncer-key | 1h |
kv/velero/minio |
velero | velero-minio | 1h |
kv/nas-ingress/vaultwarden-minio |
nas-ingress | vaultwarden-minio | 1h |
kv/nas-ingress/vaultwarden-smtp |
nas-ingress | vaultwarden-smtp | 1h |
kv/ops/terraform-cloud |
— | (reference only, not VSO) | — |
See the Guide: Adding a New Vault Secret for step-by-step instructions.