Storage in StatefulSets

A Deployment that mounts a PVC gives the same volume to all its replicas. For a database, this is catastrophic: two replicas writing to the same disk causes data corruption. StatefulSets use volumeClaimTemplates instead: a template that creates a unique, dedicated PVC for each Pod.

web-0 gets PVC data-web-0. web-1 gets PVC data-web-1. web-2 gets PVC data-web-2. When a Pod is deleted and recreated, it is reattached to its original PVC. The data persists across restarts.

Adding volumeClaimTemplates

nano statefulset-storage.yaml

apiVersion: v1
kind: Service
metadata:
  name: db-headless
spec:
  clusterIP: None
  selector:
    app: db
  ports:
    - port: 5432
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: db
spec:
  serviceName: db-headless
  replicas: 2
  selector:
    matchLabels:
      app: db
  template:
    metadata:
      labels:
        app: db
    spec:
      containers:
        - name: database
          image: busybox:1.36
          command:
            - sh
            - -c
            - |
              echo "init data for $HOSTNAME" > /data/init.txt
              sleep 3600
          volumeMounts:
            - name: data
              mountPath: /data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ['ReadWriteOnce']
        resources:
          requests:
            storage: 1Gi

kubectl apply -f statefulset-storage.yaml

What volumeClaimTemplates creates

kubectl get pvc

You will see two PVCs:

• data-db-0: created for db-0, bound to a PV
• data-db-1: created for db-1, bound to a PV

The naming convention is <template-name>-<pod-name>.

kubectl get pv

Each PVC is backed by a Persistent Volume. The StatefulSet controller did not create the PVs directly: it created the PVCs, and the PVC binding mechanism (StorageClass or pre-provisioned PVs) created or assigned the PVs.

kubectl exec db-0 -- cat /data/init.txt
kubectl exec db-1 -- cat /data/init.txt

Each Pod wrote its own hostname to its own /data/init.txt file. db-0 writes to data-db-0, db-1 writes to data-db-1. They never share a volume.

Storage and rescheduling

When a Pod is rescheduled to a different node, its PVC must be accessible from that node. This depends on the accessMode:

• ReadWriteOnce (RWO): the volume can only be mounted on one node at a time. If the Pod moves to a different node and the volume is backed by local storage (like hostPath), the Pod may get stuck. Cloud block storage (AWS EBS, GCP PD) supports ReadWriteOnce and can be remounted on a different node.
• ReadWriteMany (RWX): the volume can be mounted by multiple nodes simultaneously. Network filesystems (NFS, CephFS) typically support this.

For most StatefulSet use cases, use a StorageClass that provisions cloud block storage, which supports ReadWriteOnce across nodes.

Inspecting PVC state

kubectl describe pvc data-db-0

The output shows:

• Status: Bound: the PVC has a backing PV
• Volume: the name of the bound PV
• Access Modes: how the volume can be mounted
• StorageClass: which storage class was used to provision it

If a PVC stays in Pending, it cannot find a PV that satisfies its requirements. Check kubectl describe pvc Events for the reason.

kubectl delete statefulset db
kubectl delete service db-headless

The PVCs are retained. You would need to delete them manually.

Storage in StatefulSets is per-instance, stable, and not cleaned up automatically. This is the right behavior for databases. The next lesson covers how StatefulSets handle updates and the different update strategies available.