How to Resize a Kubernetes StatefulSet’s Volumes

Kubernetes StatefulSets are at risk of deploy stateful functions inner your cluster. Each and each Pod within the StatefulSet can access local persistent volumes that follow it even after it’s rescheduled. This allows Pods to take particular particular person issue that’s decrease free their neighbors within the distance.

Unfortunately these volumes attain with a huge limitation: Kubernetes doesn’t present a trend to resize them from the StatefulSet object. The spec.sources.requests.storage property of the StatefulSet’s volumeClaimTemplates field is immutable, preventing you from applying any capability will enhance you require. This article will indicate you workaround the anguish.

Growing a StatefulSet

Copy this YAML and place it apart to ss.yaml:

apiVersion: v1 kind: Carrier metadata:    name: nginx   labels:      app: nginx spec:    selector:      app: nginx   ports:    - name: nginx     port: 80   clusterIP: None --- apiVersion: apps/v1 kind: StatefulSet metadata:    name: nginx spec:    selector:      matchLabels:        app: nginx   replicas: 3   serviceName: nginx   template:      metadata:        labels:          app: nginx     spec:        containers:        - name: nginx         image: nginx:most modern         ports:          - name: net           containerPort: 80         volumeMounts:          - name: files           mountPath: /usr/allotment/nginx/html   volumeClaimTemplates:    - metadata:        name: files     spec:        accessModes: ["ReadWriteOnce"]       sources:          requests:            storage: 1Gi

Notice the YAML to your cluster with Kubectl:

$ kubectl prepare -f ss.yaml  carrier/nginx created  statefulset.apps/nginx created

You’ll need a storage class and provisioner to your cluster to escape this instance. It creates a StatefulSet that runs three replicas of an NGINX net server.

Whereas this isn’t advisor of when StatefulSets needs to be prone, it’s ample as a demo of the volume complications you can per chance be in a intention to face. A volume claim with 1 Gi of storage is mounted to NGINX’s files directory. Your net drawl material might per chance per chance presumably outgrow this relatively shrimp allowance as your carrier scales. Then again making an are attempting to adjust the volumeClaimTemplates.spec.sources.requests.storage field to 10Gi will file the following error when you escape kubectl prepare:

$ kubectl prepare -f ss.yaml  carrier/nginx unchanged  The StatefulSet "nginx" is invalid: spec: Forbidden: updates to statefulset spec for fields rather then 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden

This occurs on yarn of practically the general fields of a StatefulSet’s manifest are immutable after introduction.

Manually Resizing StatefulSet Volumes

You might per chance per chance presumably also bypass the restriction by manually resizing the persistent volume claim (PVC). You’ll then need to recreate the StatefulSet to open and rebind the volume out of your Pods. This might per chance per chance trigger the explicit volume resize tournament.

First exercise Kubectl to search out the PVCs associated collectively with your StatefulSet:

$ kubectl rep pvc  NAME           STATUS   VOLUME                                     CAPACITY   ACCESS MODES  files-nginx-0   Sure    pvc-ccb2c835-e2d3-4632-b8ba-4c8c142795e4   1Gi        RWO           files-nginx-1   Sure    pvc-1b0b27fe-3874-4ed5-91be-d8e552e515f2   1Gi        RWO           files-nginx-2   Sure    pvc-4b7790c2-3ae6-4e04-afee-a2e1bae4323b   1Gi        RWO

There are three PVCs on yarn of there are three replicas within the StatefulSet. Each and each Pod will get its win particular particular person volume.

Now exercise kubectl edit to alter the capability of every volume:

$ kubectl edit pvc files-nginx-0

The PVC’s YAML manifest will seem to your editor. Get the spec.sources.requests.storage field and alternate it to your sleek desired capability:

# ... spec:    sources:      requests:        storage: 10Gi # ...

Establish and cease the file. Kubectl must file that the alternate has been applied to your cluster.

persistentvolumeclaim/files-nginx-0 edited

Now repeat these steps for the StatefulSet’s remaining PVCs. Itemizing your cluster’s persistent volumes must then indicate the sleek dimension in opposition to each one:

$ kubectl rep pv  NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                 pvc-0a0d0b15-241f-4332-8c34-a24b61944fb7   10Gi       RWO            Delete           Sure    default/files-nginx-2  pvc-33af452d-feff-429d-80cd-a45232e700c1   10Gi       RWO            Delete           Sure    default/files-nginx-0  pvc-49f3a1c5-b780-4580-9eae-17a1f002e9f5   10Gi       RWO            Delete           Sure    default/files-nginx-1

The claims will take the frail dimension for now:

$ kubectl rep pvc  NAME           STATUS   VOLUME                                     CAPACITY   ACCESS MODES  files-nginx-0   Sure    pvc-33af452d-feff-429d-80cd-a45232e700c1   10Gi       RWO           files-nginx-1   Sure    pvc-49f3a1c5-b780-4580-9eae-17a1f002e9f5   10Gi       RWO           files-nginx-2   Sure    pvc-0a0d0b15-241f-4332-8c34-a24b61944fb7   10Gi       RWO

Here’s on yarn of the volume can’t be resized while Pods are peaceable the utilization of it.

Recreating the StatefulSet

Total the resize by releasing the volume claim from the StatefulSet that’s retaining it. Delete the StatefulSet however exercise the orphan cascading mechanism so its Pods dwell to your cluster. This might per chance per chance support cut downtime.

$ kubectl delete statefulset --cascade=orphan nginx  statefulset.apps "nginx" deleted

Next edit your fashioned YAML file to encompass the sleek volume dimension within the spec.sources.requests.storage file. Then exercise kubectl prepare to recreate the StatefulSet to your cluster:

$ kubectl prepare -f ss.yaml  carrier/nginx unchanged  statefulset.apps/nginx created

The sleek StatefulSet will select ownership of the previously orphaned Pods on yarn of they’ll already meet its requirements. The volumes might per chance per chance presumably rep resized at this level however customarily you’ll bask in to manually provoke a rollout that restarts your Pods:

$ kubectl rollout restart statefulset nginx

The rollout proceeds sequentially, concentrating on one Pod at a time. This ensures your carrier stays accessible for the period of.

Now your PVCs must indicate the sleek dimension:

$ kubectl rep pvc  NAME           STATUS   VOLUME                                     CAPACITY   ACCESS MODES  files-nginx-0   Sure    pvc-33af452d-feff-429d-80cd-a45232e700c1   10Gi       RWO           files-nginx-1   Sure    pvc-49f3a1c5-b780-4580-9eae-17a1f002e9f5   10Gi       RWO           files-nginx-2   Sure    pvc-0a0d0b15-241f-4332-8c34-a24b61944fb7   10Gi       RWO

Strive connecting to 1 of your Pods to bask in a look on the increased capability is viewed from inner:

$ kubectl exec -it nginx-0 bash  root@nginx-0:/# df -h /usr/allotment/nginx/html  Filesystem                                                                Dimension  Extinct Avail Utilize% Mounted on  /dev/disk/by-identification/scsi-0DO_Volume_pvc-33af452d-feff-429d-80cd-a45232e700c1  9.9G  4.5M  9.4G   1% /usr/allotment/nginx/html

The Pod’s reporting the expected 10 Gi of storage.

Summary

Kubernetes StatefulSets support you escape stateful functions in Kubernetes with persistent storage volumes which are scoped to particular particular person Pods. Then again the flexibility this permits ends when you want resize one of your volumes. Here’s a lacking operate which on the 2nd requires loads of handbook steps to be carried out in sequence.

The Kubernetes maintainers are mindful of the anguish. There’s an open operate query to kind a resolution which must finally support you provoke volume resizes by improving a StatefulSet’s manifest. This might per chance per chance also be grand faster and safer than the sizzling anguish.

One closing caveat is that volume resizes are reckoning on a storage driver that permits dynamic expansion. This operate easiest became generally available in Kubernetes v1.24 and not all drivers, Kubernetes distributions, and cloud platforms will enhance it. You might per chance per chance presumably also evaluate whether yours does by running kubectl rep sc and buying for beautiful within the ALLOWVOLUMEXPANSION column of the storage driver you’re the utilization of collectively with your StatefulSets.