How to Create EBS Volume Snapshots on EKS?

Volume snapshots are one of the useful features of Kubernetes. Kubernetes actually uses the AWS API to create a volume snapshot.

Photo by Marco Bianchetti on Unsplash

IAM Policy

You will need to an IAM policy and attach it to your Kubernetes Node’s role.

IAM Policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "ec2:DetachVolume",
                "ec2:AttachVolume",
                "ec2:DeleteSnapshot",
                "ec2:DescribeInstances",
                "ec2:DeleteTags",
                "ec2:DescribeTags",
                "ec2:DescribeSnapshotAttribute",
                "ec2:CreateTags",
                "ec2:DescribeSnapshots",
                "ec2:CreateVolume",
                "ec2:DeleteVolume",
                "ec2:CreateSnapshots",
                "ec2:DescribeVolumes",
                "ec2:CreateSnapshot"
            ],
            "Resource": "*"
        }
    ]
}

You may want to check out this post on how to manage IAM policies for EKS.

You have to install AWS EBS CSI (Container Storage Interface) Driver on your cluster:

kubectl apply -k "github.com/kubernetes-sigs/aws-ebs-csi-driver/deploy/kubernetes/overlays/stable/?ref=v1.17.0"

The parameter -k is for Kubernetes Kustomize. The Kustomize.yaml file (here) contains all of the necessary YAML files, so you don’t have to install them one by one or worry about the sequence of the YAML files.

VolumeSnapshot, VolumeSnapshotContent, and VolumeSnapshotClass

VolumeSnapshot, VolumeSnapshotContent, and VolumeSnapshotClass are not part of the Kubernetes Core API. Thereby, you will have to install their CRDs (Custom Resource Definitions)

You can install those CRDs via one Kustomize.yaml file:

kubectl apply -k 'github.com/kubernetes-csi/external-snapshotter/client/config/crd?ref=v6.2.1'

Snapshot Controller

You have to install the snapshot controller for those CRDs:

kubectl apply -k 'github.com/kubernetes-csi/external-snapshotter/deploy/kubernetes/snapshot-controller?ref=v6.2.1'

External Snapshotter vs. Snapshot Controller

The snapshot controller will be watching the Kubernetes API for VolumeSnapshot and VolumeSnapshotContent requests.

The external snapshotter will be watching the Kubernetes API for only VolumeSnapshotContent request.

The snapshot controller will create a VolumeSnapshotContent component when we initiate a VolumeSnapshot request. After that, the external snapshot is automatically triggered since it is monitoring the Kubernetes API for VolumeSnapshotContent requests.

To create an EBS Snapshot, the external snapshotter will use the Amazon API.

01-storage-class.yml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-storageclass
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer

Kubectl apply:

➜  ~ kubectl apply -f 01-storage-class.yml
storageclass.storage.k8s.io/ebs-storageclass created

➜ ~ kubectl get storageclasses
NAME               PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
ebs-storageclass   ebs.csi.aws.com         Delete          WaitForFirstConsumer   false                  37s

02-persistent-volume-claim.yml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: busybox-pvc
spec:
  storageClassName: ebs-storageclass
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

Kubectl apply:

➜  ~ kubectl apply -f 02-persistent-volume-claim.yml
persistentvolumeclaim/busybox-pvc created
➜  ~ kubectl get pvc
NAME          STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS       AGE
busybox-pvc   Pending                                      ebs-storageclass   37s

03-pod.yml

apiVersion: v1
kind: Pod
metadata:
  name: busybox-pod
spec:
  containers:
  - name: busybox-container
    image: busybox:latest
    command:
      - sleep
      - "9999"
    volumeMounts:
    - mountPath: /usr/src/app
      name: busybox-volume
  volumes:
  - name: busybox-volume
    persistentVolumeClaim:
      claimName: busybox-pvc

Kubectl apply:

➜  ~ kubectl apply -f 03-pod.yml
pod/busybox-pod created
➜  ~ kubectl describe pods
Name:         busybox-pod
Namespace:    default
...
Volumes:
  busybox-volume:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)

SnapshotClass

04-snapshot-class.yml

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: ebs-snapshotclass
driver: ebs.csi.aws.com
deletionPolicy: Delete

Kubectl apply:

➜  ~ kubectl apply -f 04-snapshot-class.yml
volumesnapshotclass.snapshot.storage.k8s.io/ebs-snapshotclass created
➜  ~ kubectl get volumesnapshotclasses.snapshot.storage.k8s.io
NAME                DRIVER            DELETIONPOLICY   AGE
ebs-snapshotclass   ebs.csi.aws.com   Delete           57s

VolumeSnapshot

05-volume-snapshot.yml

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: first-snapshot
spec:
  volumeSnapshotClassName: ebs-snapshotclass
  source:
    persistentVolumeClaimName: busybox-pvc

Kubectl apply:

➜ ~ kubectl apply -f 05-volume-snapshot.yml
volumesnapshot.snapshot.storage.k8s.io/first-snapshot created
➜ ~ # wait a little...
➜ ~ kubectl get volumesnapshot
NAME             READYTOUSE   SOURCEPVC     SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS       SNAPSHOTCONTENT                                    CREATIONTIME   AGE
first-snapshot   true         busybox-pvc                           10Gi          ebs-snapshotclass   snapcontent-1390a21c-ca5e-436e-8db1-f751c6e26346   104s           104s
➜ ~ kubectl get volumesnapshotcontent
NAME                                               READYTOUSE   RESTORESIZE   DELETIONPOLICY   DRIVER            VOLUMESNAPSHOTCLASS   VOLUMESNAPSHOT   VOLUMESNAPSHOTNAMESPACE   AGE
snapcontent-1390a21c-ca5e-436e-8db1-f751c6e26346   true         10737418240   Delete           ebs.csi.aws.com   ebs-snapshotclass     first-snapshot   default                   106s

You can see the created snapshot on AWS EC2 Management Console:

How Can I View Logs of Snapshot Requests?

You may want to look at the logs of the snapshot controller:

kubectl logs -l app=snapshot-controller -n kube-system --all-containers

You may want to look at the EBS CSI Controller’s logs:

kubectl logs -l app=ebs-csi-controller -n kube-system --all-containers