Terminology Confusion in Kubernetes: StorageClass, PersistentVolume, PersistentVolumeClaim
There are several terms associated with storing persistent data in Kubernetes: StorageClass, PersistentVolume, PersistentVolumeClaim…
StorageClass
Imagine that as a system administrator, you want to offer multiple storage types to the developers: AWS EBS, Ceph, Cinder, etc.
You must install the driver for every storage type you want to use in your Kubernetes cluster.
After that, you must inform your cluster, “Hey, you may utilize this storage with this driver.”. To be able to inform your cluster, you must use Kubernetes’ StorageClass component.
The StorageClass component may have been called “Add a Storage Type to the Cluster”
Here’s an example of how to add AWS EBS storage to a Kubernetes cluster:
01-storageclass.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: ebs-storageclass
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
Before adding the EBS storage to the cluster, you must install the EBS Storage’s drivers on the cluster:
kubectl apply -k "github.com/kubernetes-sigs/aws-ebs-csi-driver/deploy/kubernetes/overlays/stable/?ref=master"
More details are here
Amazon EBS storage is now recognized by your cluster. The cluster will match the provisioner in the YAML file and the driver you installed.
There are available two binding modes: Immediate (default binding mode) and WaitForFirstConsumer
WaitForFirstConsumer binding mode is available for cloud-based storage (AWS, GCP, Azure).
WaitForFirstConsumer mode: A Persistent Volume will be provisioned when a Pod attempts to use the Persistent Volume Claim.
Immediate mode: A Persistent Volume will be provisioned when a Persistent Volume Claim is created.
Persistent Volume? Persistent Volume Claim? What are those?
Imagine that you have storage with 15 TB of space. You will need to run 3 different projects on the storage. You can split your storage into 3 different persistent volumes, and each project can have 5 TB of disk space at most.
The developer of each project can use 5 TB of disk space for their DTAP environments. They will have to use the Persistent Volume Claim component to ask for the disk space they need. This is static provisioning.
A lot of clusters on the cloud tend to use dynamic provisioning, as it is a flexible way to provide sufficient disk space.
For dynamic provisioning, you don’t need to create a Persistent Volume manually. The developer of each project can request disk space via the Persistent Volume Claim component. Kubernetes will create a persistent volume for you automatically. They must define the storage class they want to use.
(Note: Before creating this persistent volume claim, I created the storage class with Immediate binding mode.)
02-persistent-volume-claim.yml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ebs-database-pv-claim
spec:
storageClassName: ebs-storageclass
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 33Gi
Let’s test it:
➜ ~ kubectl create -f 02-persistent-volume-claim.yml
persistentvolumeclaim/ebs-database-pv-claim created
➜ ~ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
ebs-database-pv-claim Bound pvc-73fb8137-6b9d-432a-8db4-5a907981314a 33Gi RWO ebs-storageclass 5s➜ ~ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-73fb8137-6b9d-432a-8db4-5a907981314a 33Gi RWO Delete Bound default/ebs-database-pv-claim ebs-storageclass 28s
EBS is created by Kubernetes:
Kubernetes created an EBS disk and a persistent volume for us. After that, the persistent volume claim component was created and linked to the persistent volume.
How to use Persistent Volume and Persistent Volume Claim components together?
Since the persistent volume needs static provisioning, you will have to create an EBS disk by hand before using the persistent volume:
aws ec2 create-volume --size 15 --availability-zone eu-west-1b
(Note: The EBS disk must be in the same availability zone as your Kubernetes node)
Take note of the VolumeId in the output.
03-persistent-volume.yml
apiVersion: v1
kind: PersistentVolume
metadata:
name: ebs-database-pv
spec:
storageClassName: ebs-storageclass
capacity:
storage: 41Gi
accessModes:
- ReadWriteOnce
csi:
driver: ebs.csi.aws.com
fsType: ext4
volumeHandle: "EBS_VOLUME_ID"
Create a persistent volume:
➜ ~ kubectl create -f 03-persistent-volume.yml
persistentvolume/ebs-database-pv created
➜ ~ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
ebs-database-pv 41Gi RWO Retain Available ebs-storageclass 7s
You can create a persistent volume claim in this persistent volume:
04-persistent-volume-claim.yml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ebs-database-pv-claim
spec:
storageClassName: ebs-storageclass
volumeName: ebs-database-pv
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 33Gi
Create a persistent volume claim and check it:
➜ ~ kubectl create -f 04-persistent-volume-claim.yml
persistentvolumeclaim/ebs-database-pv-claim created
➜ ~ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
ebs-database-pv 41Gi RWO Retain Bound default/ebs-database-pv-claim ebs-storageclass 110s
➜ ~ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
ebs-database-pv-claim Bound ebs-database-pv 41Gi RWO ebs-storageclass 22s
Please notice that we requested a smaller size than what was available in the persistent volume and received the same amount of available size in the persistent volume.
Try to attach multiple volume claims to one persistent volume
Let’s try to create another volume claim on the same persistent volume:
05-persistent-volume-claim-another.yml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ebs-database-pv-claim-another
spec:
storageClassName: ebs-storageclass
volumeName: ebs-database-pv
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
➜ ~ kubectl create -f 05-persistent-volume-claim-another.yml
persistentvolumeclaim/ebs-database-pv-claim-another created
➜ ~ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
ebs-database-pv-claim Bound ebs-database-pv 41Gi RWO ebs-storageclass 4m15s
ebs-database-pv-claim-another Pending ebs-database-pv 0 ebs-storageclass 4s
ebs-database-pv-claim-another
is in the “Pending” status. Let’s see why:
➜ blog kubectl describe pvc ebs-database-pv-claim-another
Name: ebs-database-pv-claim-another
Namespace: default
StorageClass: ebs-storageclass
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedBinding 3s (x7 over 82s) persistentvolume-controller volume "ebs-database-pv" already bound to a different claim.
A persistent volume and a persistent volume claim have a one-to-one relationship. If you need another persistent volume claim, you will require another persistent volume.
Request a larger disk space than what is available
Let’s try to create a persistent volume claim (100 Gib) that has a larger size than what is available in the persistent volume (41 Gib).
06-persistent-storage-claim.yml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ebs-database-pv-claim
spec:
storageClassName: ebs-storageclass
volumeName: ebs-database-pv
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100Gi
(Note: Before executing this yaml file, I deleted the persistent volume.)
Create it:
➜ ~ kubectl create -f 03-persistent-volume.yml
persistentvolume/ebs-database-pv created
➜ ~ kubectl create -f 06-persistent-storage-claim.yml
persistentvolumeclaim/ebs-database-pv-claim created
➜ ~ kubectl describe pvc
Name: ebs-database-pv-claim
Namespace: default
StorageClass: ebs-storageclass
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning VolumeMismatch 0s (x2 over 11s) persistentvolume-controller Cannot bind to requested volume "ebs-database-pv": requested PV is too smallKubernetes didn’t allow me to use a larger size than what is available in the persistent volume.