Storage and CSI

Key points:

• Always use volumeBindingMode: WaitForFirstConsumer so disks are created in the pod’s zone.
• Block volumes are zonal: a pod with an EBS volume can only run in that volume’s zone.
• RWO means one node, not one pod. Use RWOP for a single writer.
• reclaimPolicy: Delete deletes the disk with the PVC. Use Retain plus backups for data you care about.
• Pick block (fast, zonal) or file (shared, slower) storage by access pattern, not habit.

The pieces

Object	Who creates it	What it is
StorageClass	Platform team	A recipe: which driver, which disk type, binding and reclaim rules
PersistentVolumeClaim (PVC)	The app	A request: “10 GiB, ReadWriteOnce, from this StorageClass”
PersistentVolume (PV)	Usually the driver	The actual disk, bound one-to-one to a claim
VolumeAttachment	Kubernetes	Records that a disk is attached to a node

The CSI driver (Container Storage Interface) talks to the storage API (EBS, EFS, Azure Disk, Persistent Disk) in two parts:

• Controller (a Deployment): creates, deletes, attaches, resizes and snapshots volumes. Sidecars (external-provisioner, external-attacher) watch Kubernetes objects and call the driver.
• Node plugin (a DaemonSet on every node): formats and mounts volumes for the kubelet to hand to pods.

From claim to mounted disk

sequenceDiagram
    participant App as PVC
    participant Prov as CSI controller
    participant Sched as Scheduler
    participant Att as Attach (controller)
    participant Kubelet as Kubelet + CSI node plugin
    App->>Prov: 1. PVC created
    Prov->>Prov: 2. CreateVolume (cloud API), PV bound
    Sched->>Sched: 3. Pod scheduled to a node
    Att->>Att: 4. ControllerPublishVolume: attach disk to that node
    Kubelet->>Kubelet: 5. NodeStageVolume: format and mount once per node
    Kubelet->>Kubelet: 6. NodePublishVolume: bind-mount into the pod

1. Provision. A new PVC triggers CreateVolume - a real cloud disk - and a PV is bound to the claim.
2. Schedule. The pod lands on a node that can reach the volume.
3. Attach. Block storage (EBS) is attached to the node’s instance; file storage (EFS, NFS) skips this.
4. Stage. The node plugin formats a new disk and mounts it at a global node path.
5. Publish. It bind-mounts that path into the pod; pods on one node can share one staged mount.

Access modes

Pick by access pattern, not habit: block storage is fast and zonal, file storage is shared and slower.

Mode	Means	Typical backend
ReadWriteOnce (RWO)	Read-write from one node - several pods on that node can share it	EBS, Azure Disk, Persistent Disk
ReadWriteOncePod (RWOP)	Read-write from exactly one pod	Same block storage, stricter
ReadOnlyMany (ROX)	Read-only from many nodes	Snapshots, shared config
ReadWriteMany (RWX)	Read-write from many nodes	EFS, Azure Files, Filestore, NFS

Gotchas

• Immediate binding picks the wrong zone. The disk is created before scheduling, possibly in a zone the pod can’t run in. Use volumeBindingMode: WaitForFirstConsumer.
• Block volumes are zonal. An EBS volume in eu-central-1a can never attach in 1b, so its StatefulSet pod can only run there. If the zone fails, so does the pod.
• RWO means one node, not one pod. Two pods on the same node can both write. Use RWOP when you need exactly one writer.
• Multi-attach errors after a node dies. The disk stays attached to the dead node for several minutes before force-detach. The replacement pod sits in ContainerCreating until then.
• reclaimPolicy: Delete deletes the disk. Deleting the PVC deletes the disk. For data you care about, use a Retain StorageClass plus backups.
• StatefulSet volumes outlive their pods. Scaling down keeps the PVCs, and so the bill. Set persistentVolumeClaimRetentionPolicy to have them removed.
• fsGroup on big volumes is slow. By default Kubernetes chowns every file at mount time. Set fsGroupChangePolicy: OnRootMismatch.
• subPath mounts never update. A Secret or ConfigMap mounted with subPath keeps its old content forever. Mount the directory instead.
• Resizing is opt-in and grow-only. Volumes can only grow, never shrink. Set allowVolumeExpansion: true on the StorageClass.
• Per-pod network file mounts cost memory. Hundreds of pods each mounting EFS can eat gigabytes in the CSI driver. Use a one-mount-per-node pattern - see taming EFS memory bloat.

A sensible StorageClass

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3-retain
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  encrypted: "true"
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Retain
allowVolumeExpansion: true