2026-05 update: Longhorn was replaced by Rook-Ceph (2026-05-17) and the
TrueNAS NFS tier was decommissioned (2026-05-24). All formerlonghorn,
nfs-subdir-retain, andnfs-subdir-deletePVCs now live on Ceph.
Three storage tiers, each optimised for a different workload type.
| Class | Backend | Media | Access | Reclaim | Best For |
|---|---|---|---|---|---|
ceph-block (default) |
Rook-Ceph RBD | NVMe SSD, 2-replica | RWO | Retain | Databases, all critical stateful apps |
ceph-filesystem |
Rook-Ceph CephFS | NVMe SSD, 2-replica | RWX | Retain | Shared blobs, config, logs (in-cluster RWX) |
nfs-synology |
Synology 192.168.88.19 | 4×8TB HDD | RWX | Retain | Large user files / media, bulk capacity |
Large media volumes (Immich 10Ti, Jellyfin 2Ti, Kavita 500Gi) use direct NFS
PersistentVolumes to the Synology rather than a dynamic StorageClass.
Retain — PV stays after the PVC is deleted; data is safe but must be cleaned up manually. All three classes use Retain.Rook-Ceph replaced Longhorn on 2026-05-17. It runs entirely in-cluster and pools
the NVMe-backed block device attached to each of the six worker VMs into a single
distributed cluster.
| Setting | Value |
|---|---|
| Operator chart | rook-ceph v1.20.x (sync wave 2) |
| Cluster CRs | core-components/rook-ceph/resources/ (wave 5–7) |
| Ceph version | quay.io/ceph/ceph:v20.2.2 (Tentacle) |
| Namespace | rook-ceph |
| Dashboard | https://rook-ceph.homelab.vyanh.uk |
| OSDs | 6 (one per worker, talos-worker1–6) |
| MONs / MGRs | 3 MON, 2 MGR |
| MDS | CephFS metadata server, active + standby |
| Raw capacity | ~3.4 TB (sum of all OSD devices) |
| Replication | 2 copies, failure domain = host |
All OSD disks are NVMe SSD passed through QEMU virtio-blk, which reports
rotational=1in sysfs. Ceph would auto-classify them as HDD, so
crushDeviceClass: "ssd"is pinned in the CephCluster spec.
The dashboard (and the Uptime-Kuma "Rook-Ceph" monitor) flapped with 503s roughly
daily: the active mgr leaked ~200–250 MB/h until OOMKilled (2Gi limit → 7
restarts on 2026-07-06; raising to 4Gi only stretched the period to ~18 h).
Root cause: the rook orchestrator mgr module's embedded kubernetes python
client fetched the full rook-ceph PodList (~1 MB) every 15 s and logged the whole
response body at DEBUG into the mgr in-memory log ring (log_max_recent
10000 → up to ~10 GB retained). No ceph config knob reaches that library logger —
debug_mgr, mgr/rook/log_level and mgr restarts all failed to silence it.
Fix (in core-components/rook-ceph/resources/cephcluster.yaml):
spec.mgr.modules: rook module enabled: false. The Rook operator managesceph orch CLI (now errors — expected)spec.cephConfig.mgr.log_max_recent: "500" as a bounded-ring safety net.Live-apply without waiting for a mgr rebuild: ceph orch set backend '' && ceph mgr module disable rook. Verify: mgr container log volume drops ~270×, and
container_memory_working_set_bytes{pod=~"rook-ceph-mgr.*",container="mgr"} goes flat.
| OSD node | Host | Device | Size |
|---|---|---|---|
| talos-worker1 | pve1 | /dev/ceph-data/worker1 (LVM) |
415 GB |
| talos-worker2 | pve1 | /dev/ceph-data/worker2 (LVM) |
415 GB |
| talos-worker4 | pve2 | ZFS zvol | 1000 GB |
| talos-worker5 | pve2 | ZFS zvol | 550 GB |
| talos-worker3 | pve3 | ZFS zvol | 300 GB |
| talos-worker6 | pve3 | ZFS zvol | 550 GB |
Default StorageClass. Replicated block volumes for databases and any
single-writer stateful workload.
replicapool, size: 2, failureDomain: host, compression_mode: aggressivevolumeBindingMode: WaitForFirstConsumer, fstype: xfsrook-ceph.rbd.csi.ceph.comIn-cluster ReadWriteMany — the replacement for the old TrueNAS nfs-subdir-retain
tier. Used where multiple pods/nodes need shared write access without depending
on the NAS.
cephfs: metadata pool size: 3, data pool size: 2 (compression_mode: aggressive)rook-ceph.cephfs.csi.ceph.com⚠️ Talos requires the FUSE mounter. Talos ships a minimal kernel without
thecephkernel module, so kernel CephFS mounts fail with
modprobe: FATAL: Module ceph not found. The StorageClass must set
mounter: fusein itsparameters. StorageClass parameters are immutable
after creation — to change the mounter you must delete and recreate the
StorageClass, then delete + reprovision any PVCs created under the old params.
volumeClaimTemplates are immutable, so migrating a StatefulSet between classes
requires recreating the PVC:
# 1. Disable ArgoCD auto-sync so it doesn't fight you
kubectl patch application <name> -n argocd --type=json \
-p '[{"op":"remove","path":"/spec/syncPolicy/automated"}]'
# 2. Dump the database FIRST (if PostgreSQL)
kubectl exec -n <ns> <postgres-pod> -- pg_dump -U postgres <db> > backup.sql
# 3. Delete the StatefulSet WITHOUT deleting pods (orphan keeps them running)
kubectl delete sts <name> -n <ns> --cascade=orphan
# 4. Delete the old PVC, then the pod
kubectl delete pvc <pvc-name> -n <ns>
kubectl delete pod <pod-name> -n <ns>
# 5. Re-enable ArgoCD auto-sync — it recreates the STS with the new class
# 6. Restore the database
kubectl cp backup.sql <ns>/<new-pod>:/tmp/restore.sql
kubectl exec -n <ns> <new-pod> -- psql -U postgres -f /tmp/restore.sql
| Setting | Value |
|---|---|
| StorageClass | nfs-synology |
| Reclaim Policy | Retain |
| Access Mode | ReadWriteMany |
| Chart | nfs-subdir-external-provisioner v4.0.18 |
| Namespace | nfs-storage, sync wave 3 |
| NFS Server | 192.168.88.19 (Synology DS920+, 4×8TB HDD) |
| Protocol | NFSv4 |
The nfs-subdir-external-provisioner creates a subdirectory
<namespace>-<pvcname>-<pvname>/ per PVC on a shared NFS export and mounts it as
the PV. All volumes on the class share one export — only the subdir differs,
which is why RWX works. On deletion with Retain, the subdir is renamed with an
archived- prefix (not removed).
Synology's NFS uses root_squash, remapping root (uid 0) to uid 1024 (the
Synology admin user). Any container writing to this NFS mount as root is
remapped to uid 1024.
# All pods must run as the NFS owner
podSecurityContext:
runAsUser: 1024
runAsGroup: 1024
fsGroup: 1024
# Disable chown init containers — they run as root and fail under root_squash
initChownData:
enabled: false
# For pods binding port 80 (e.g. Nextcloud), root_squash blocks privileged ports:
podSecurityContext:
sysctls:
- name: net.ipv4.ip_unprivileged_port_start
value: "80"
# First deploy rsyncs many small files and is slow — use generous startup probes
startupProbe:
failureThreshold: 60
periodSeconds: 10
Directories on Synology BtrFS volumes carry + ACLs (drwxrwxrwx+) that aren't
visible via POSIX chmod/chown. Inside a Docker container they collapse to
dr-xr-xr-x (read-only) for non-owner UIDs, causing EACCES even when host
perms look fine.
/var/lib/docker/volumes/) for any directory a container must write to — they bypass Synology ACLs.cpus:/cpu_shares: cause NanoCPUs can not be set. Use only mem_limit/mem_reservation.MinIO runs as a Docker container on the Synology NAS (192.168.88.19:9000) and
exposes an S3-compatible API. It is the backup/object backend, not a K8s
StorageClass.
| Service | How MinIO is Used |
|---|---|
| Velero | K8s backup manifests (JSON) + Kopia PVC data chunks |
| Harbor | Image layer storage — each OCI layer as an S3 object |
| Tempo | Trace span storage |
| pg_dump | One gzipped SQL file per database per day |
| MikroTik | Daily router config export (FTP) |
Why MinIO instead of Synology's native S3? Synology's S3 implementation has
compatibility issues with some clients; MinIO is fully S3-compatible and
well-tested.
| Bucket | User | Consumer |
|---|---|---|
velero-backups |
velero |
Velero K8s + PVC backups |
tempo-traces |
tempo |
Tempo distributed traces |
harbor-registry |
harbor-registry |
Harbor container image blobs |
db-backups |
db-backup |
pg_dump CronJobs (all DB namespaces) |
router-backups |
mikrotik |
MikroTik RouterOS config exports |
terraform-state |
(TF backend) | terraform-k8s-infra state |
Each user has a least-privilege policy scoped to its own bucket only.
disableredirect: trueHarbor normally redirects clients to MinIO via a presigned URL, but the redirect
host (192.168.88.19) isn't reachable from outside the cluster. Setting
disableredirect: true makes Harbor proxy the data itself — slightly slower but
works from any network.
# mc lives inside the container — no local install
alias mc='ssh 192.168.88.19 PATH=/usr/local/bin:/usr/bin:/bin docker exec minio mc'
mc du local/ # bucket sizes
mc ls local/velero-backups/backups/
| App | StorageClass | Access | Notes |
|---|---|---|---|
| Vault data/audit (×3 each) | ceph-block |
RWO | Critical secrets + audit |
| Authentik / Harbor / Nextcloud / LifeOps / Vikunja / Mealie / Ente / Firefly PostgreSQL | ceph-block |
RWO | App databases |
| VictoriaMetrics / VictoriaLogs / Tempo | ceph-block |
RWO | Monitoring time-series |
| Technitium (×3), Omada+Mongo, Harbor Redis, Nextcloud Redis | ceph-block |
RWO | Stateful singletons |
| Harbor Trivy / Harbor JobService | ceph-filesystem |
RWX | Migrated off TrueNAS 2026-05-24 |
| Grafana | ceph-filesystem |
RWX | Dashboards/plugins |
| Vector GeoIP DB | ceph-filesystem |
RWX | Shared GeoIP data |
| Vaultwarden LiteFS | ceph-filesystem |
RWX | K8s warm-standby replica |
| Nextcloud user files / nas-home | nfs-synology |
RWX | Bulk HDD capacity |
| Frigate media (1Ti), CrowdSec config/db, Home Assistant config | nfs-synology |
RWX | Bulk HDD |
| Immich photos (10Ti), Jellyfin media (2Ti), Kavita library (500Gi) | direct NFS PV | RWX | Large media, Synology |
| Scenario | Recommended | Reasoning |
|---|---|---|
| PostgreSQL / databases | ceph-block |
Consistent IOPS, replicated, no NFS latency spikes |
| Monitoring time-series | ceph-block |
Block performance, stays up independent of NAS |
| In-cluster shared RWX (config, logs, blobs) | ceph-filesystem |
RWX without a NAS dependency (FUSE mounter on Talos) |
| Large user files / media | nfs-synology (or direct NFS PV) |
HDD capacity >> performance |
| Object storage (backups, traces) | MinIO | Not a StorageClass — use the S3 API directly |