The terraform-k8s-infra repository manages Vault infrastructure (unseal, authentication, secrets) and Harbor configuration using Terraform. State lives in NAS MinIO (migrated from Terraform Cloud on 2026-04-28).
No longer Terraform Cloud. State was migrated to NAS MinIO on 2026-04-28 to remove the external dependency, simplify CI, and let backups land in the same S3-compatible store as everything else. The Terraform Cloud workspace still exists but is unused; the
TF_API_TOKENCI variable was deleted.
| Setting | Value |
|---|---|
| Backend | s3 (against MinIO) |
| Endpoint | http://192.168.88.19:9000 (MinIO on NAS) |
| Bucket | terraform-state |
| Key | k8s-infra-lab/terraform.tfstate |
| Region | us-east-1 (placeholder — MinIO ignores it but the AWS provider requires one) |
| Locking | DynamoDB-style locking via MinIO is not enabled — single-developer use, locking handled by GitLab CI runner concurrency |
backend.tf:
terraform {
backend "s3" {
bucket = "terraform-state"
key = "k8s-infra-lab/terraform.tfstate"
region = "us-east-1"
endpoints = { s3 = "http://192.168.88.19:9000" }
skip_credentials_validation = true
skip_metadata_api_check = true
skip_region_validation = true
skip_requesting_account_id = true
use_path_style = true
}
}
Required GitLab CI variables (project-level on terraform-k8s-infra):
| Variable | Purpose |
|---|---|
MINIO_TF_ACCESS_KEY |
MinIO user with readwrite on terraform-state bucket |
MINIO_TF_SECRET_KEY |
(matching secret key) |
The CI job exports these as AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY so the AWS Terraform provider's S3 backend picks them up.
Local terraform plan from a workstation: export the same variables and run terraform init -reconfigure after pulling. There is no .terraformrc to manage anymore.
MinIO must stay up for
terraform planto work. This is fine in practice (MinIO is on NAS, very high uptime), but if you ever need to rebuild MinIO from scratch — restore theterraform-statebucket from MinIO backup first, then re-run Terraform.
| Setting | Value |
|---|---|
| Organization | TA |
| Workspace | k8s-infra-lab |
| Workspace ID | ws-dcvtGTYCdz1149vb |
| Backend | Terraform Cloud (no local state) |
Credentials (stored at kv/ops/terraform-cloud in Vault):
tfc_api_token — TFC user API tokenvault_terraform_token — scoped orphan Vault token (terraform-k8s-infra policy)Workspace variables:
vault_token (sensitive) — orphan periodic Vault token, policy: terraform-k8s-infravault_unseal_key_1/2/3 (sensitive) — Vault recovery keys for the unseal jobharbor_username, harbor_password, harbor_oidc_client_secret — Harbor credentialsgithub_token — PAT for syncing Harbor robot credentials to GitHub Actionsmodules/vault-unseal/)Automatically unseals Vault pods after cluster restarts using a Kubernetes Job.
| Setting | Value |
|---|---|
| Vault Address | https://vault.vault.svc:8200 |
| Pod Count | 3 (HA Raft cluster) |
| Unseal Keys | 3 keys via Terraform Cloud variables |
| Image | curlimages/curl:8.10.1 |
| Re-run Trigger | force_run = timestamp() (every apply) |
| Timeout | 300s active deadline, 300s TTL after finish |
How it works:
The module uses ignore_changes = [data] on the secret to prevent Terraform from reverting manually-updated keys.
modules/vault-k8s-auth-vso/)Creates per-namespace Vault policies and Kubernetes auth roles for the Vault Secrets Operator. Also manages the KV v2 mount.
Configured Namespaces (15):
monitoring, cloudflared-connector, mimir, authentik, argocd, harbor, life-ops, nextcloud, pihole, velero, technitium, external-dns, crowdsec, traefik, nas-ingress
For each namespace, this creates:
| Resource | Name | Purpose |
|---|---|---|
| Policy | vso-<namespace>-read |
Read kv/data/<namespace>/* and kv/metadata/<namespace>/* |
| Role | vso-<namespace> |
Bound to SA vso-auth in namespace, TTL 3600s |
KV mount management:
kv/ KV v2 mount (create_kv_mount = true)max_versions = 10 via vault_kv_secret_backend_v2 to prevent unbounded storage growthReviewer credential resolution:
vault-server-token Secret in vault namespacemodules/vault-secrets-placeholder/)Seeds Vault with placeholder secrets (depends on vso_k8s_per_ns to ensure policies exist first).
| Secret | Vault Path | Keys |
|---|---|---|
| Grafana Admin | kv/monitoring/grafana/admin |
username, password |
| Cloudflared Tunnel | kv/cloudflared-connector/tunnel |
token |
| Authentik Secret Key | kv/authentik/secrets |
secret-key |
| Authentik PostgreSQL | kv/authentik/postgresql |
password, postgres-password |
| ArgoCD OIDC | kv/authentik/argocd-oidc |
clientSecret |
| ArgoCD OIDC (local) | kv/argocd/oidc |
clientSecret |
| Grafana OIDC | kv/authentik/grafana-oidc |
clientSecret |
| Grafana OIDC (local) | kv/monitoring/grafana-oidc |
clientSecret |
| Harbor OIDC | kv/authentik/harbor-oidc |
clientSecret |
| Harbor OIDC (local) | kv/harbor/oidc |
clientSecret |
| GitLab OIDC | kv/authentik/gitlab-oidc |
clientSecret |
| GitLab Admin | kv/gitlab/admin |
username, password |
| Vaultwarden Admin | kv/vaultwarden/admin |
password, token_hash |
| Cloudflare API Token | kv/cloudflare/api-token |
token |
| ArgoCD LifeOps Repo | kv/argocd/repos/lifeops |
username, password |
| Nextcloud Admin | kv/nextcloud/admin |
nextcloud-username, nextcloud-password |
| Pi-hole Admin | kv/pihole/admin |
password |
| Technitium Admin | kv/technitium/admin |
password |
| Technitium TSIG | kv/technitium/tsig |
secret |
| ExternalDNS TSIG | kv/external-dns/tsig |
secret |
| CrowdSec Bouncer | kv/crowdsec/bouncer |
api_key |
kv/traefik/crowdsec-bouncer |
Removed after Istio migration — bouncer key now lives only at kv/crowdsec/bouncer |
|
| LifeOps Secrets | kv/life-ops/secrets |
DB_PASSWORD, TELEGRAM_BOT_TOKEN, TELEGRAM_CHAT_ID |
| Alertmanager Telegram | kv/monitoring/alertmanager/telegram |
bot-token |
| Tempo MinIO | kv/monitoring/tempo-minio |
AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY |
| Velero MinIO | kv/velero/minio |
cloud |
| Harbor Registry S3 | kv/harbor/registry-s3 |
REGISTRY_STORAGE_S3_ACCESSKEY, REGISTRY_STORAGE_S3_SECRETKEY |
| DB Backup MinIO (×4) | kv/<ns>/db-backup-minio |
access_key, secret_key |
| Vaultwarden warm-standby | kv/nas-ingress/vaultwarden-minio |
access_key, secret_key |
| Vaultwarden SMTP | kv/nas-ingress/vaultwarden-smtp |
SMTP_PASSWORD |
| Terraform Cloud | kv/ops/terraform-cloud |
tfc_api_token, vault_terraform_token, org, workspace |
All placeholder values are "PLACEHOLDER_UPDATE_IN_VAULT". The module uses ignore_changes = [data_json] to prevent Terraform from reverting manually-set real values.
After terraform apply, update real values:
vault kv patch kv/<namespace>/<secret> key='REAL_VALUE'
modules/harbor/)Manages Harbor container registry configuration via the Harbor Terraform provider.
| Setting | Value |
|---|---|
| Registry Host | harbor.homelab.vyanh.uk |
| OIDC Provider | Authentik |
| OIDC Endpoint | https://authentik.homelab.vyanh.uk/application/o/harbor/ |
| OIDC Client ID | harbor |
| OIDC Scopes | openid,email,profile,groups |
| OIDC Admin Group | Harbor Admins |
| Auto-Onboard | true |
Robot Accounts:
| Account | Permissions | K8s Secret | Namespace |
|---|---|---|---|
| GHA Robots | Push + Pull | harbor-gha-robots-dockercfg |
gha-runners |
| Applications Pull | Pull only | harbor-registry-secrets |
life-ops, vault-ui |
GitHub Actions secret sync:
The module automatically syncs Harbor robot credentials to GitHub Actions secrets HARBOR_USERNAME and HARBOR_PASSWORD in the LifeOps repo on every apply (requires github_token workspace variable).
Terraform uses a scoped orphan token (not root). The terraform-k8s-infra policy grants:
# Auth backends
path "sys/auth" { capabilities = ["read", "list"] }
path "sys/auth/*" { capabilities = ["create", "read", "update", "delete", "list", "sudo"] }
# Auth backend config/roles
path "auth/*" { capabilities = ["create", "read", "update", "delete", "list"] }
# Secret engine mounts
path "sys/mounts" { capabilities = ["read", "list"] }
path "sys/mounts/*" { capabilities = ["create", "read", "update", "delete", "list", "sudo"] }
# Policies
path "sys/policies/acl" { capabilities = ["list"] }
path "sys/policies/acl/*" { capabilities = ["create", "read", "update", "delete", "list"] }
path "sys/policy" { capabilities = ["list"] }
path "sys/policy/*" { capabilities = ["create", "read", "update", "delete", "list"] }
# KV v2 secrets
path "kv/*" { capabilities = ["create", "read", "update", "delete", "list", "patch"] }
Recreating the Terraform token (when it expires or is invalid):
# Generate root token (emergency procedure — see vault-vso.md)
ROOT="hvs.xxxx"
# Create orphan periodic token (MUST use -orphan so it survives root revocation)
kubectl exec -n vault vault-0 -- sh -c "VAULT_SKIP_VERIFY=1 VAULT_TOKEN=$ROOT \
vault token create -policy=terraform-k8s-infra -display-name=terraform-k8s-infra \
-orphan -period=768h -format=json"
# Update TFC workspace variable via API
curl -X PATCH https://app.terraform.io/api/v2/workspaces/ws-dcvtGTYCdz1149vb/vars/<var-id> \
--header "Authorization: Bearer $TFC_TOKEN" \
--header "Content-Type: application/vnd.api+json" \
--data '{"data":{"type":"vars","attributes":{"value":"<new-token>","sensitive":true}}}'
# Revoke root token when done
vault token revoke -self
cd envs/lab
terraform init
terraform plan
terraform apply
Uses Terraform Cloud backend — no local state file.
envs/lab/main.tfnamespaces list in module "vso_k8s_per_ns":namespaces = [..., "NEW-NAMESPACE"]
vso-NEW-NAMESPACE-read and role vso-NEW-NAMESPACEsa.yaml, vault-auth.yaml, vaultstaticsecret.yaml in k8s-cluster-configSee Guide: Adding a New Vault Secret for the full flow.