Velero: Kubernetes Backup and Restore Solution for AWS EKS

In my previous article, I discussed the installation of Velero on Azure AKS, including taking backups of Azure AKS and restoring them to another Azure AKS cluster in the event of a cluster crash. Now, I will demonstrate how to install Velero on AWS EKS, take backups of AWS EKS, and restore them to another AWS EKS cluster.

Velero: Kubernetes Backup and Restore Solution for Azure AKS

Step-1: First, In the application I will create on AWS EKS, I will use a dynamic volume for persistent volume claims and expose the application to external traffic. To obtain a certificate from Let’s Encrypt, I will set up cert-manager. I will deploy ingress-nginx to acquire an external IP and expose the application. Subsequently, I will create the necessary records in my DNS provider, Route 53, to make my host publicly accessible.

# creating namespaces
kubectl create ns cert-manager
kubectl create ns nginx

# adding and update helm-charts
helm repo add jetstack https://charts.jetstack.io
helm repo add nginx https://kubernetes.github.io/ingress-nginx
helm repo update

# install cert-manager
helm upgrade --install cert-manager jetstack/cert-manager --create-namespace --namespace cert-manager --set installCRDs=true

# note: Before installing ingress-nginx, create a helm-nginx.yaml file locally.
# helm-nginx.yaml
controller:
  service:
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-subnets: subnet-0exxxxxxxxxxxxx # public subnet id
      service.beta.kubernetes.io/aws-load-balancer-type: nlb
      service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
      service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing

# install ingress-nginx
helm upgrade --install ingress-nginx nginx/ingress-nginx --create-namespace --namespace nginx -f helm-nginx.yaml

# apply cert-cluster-issuer.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    email: mehmetkanus17@gmail.com
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: issuer-account-key
    solvers:
      - http01:
          ingress:
            class: nginx
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    email: mehmetkanus17@gmail.com
    server: 'https://acme-v02.api.letsencrypt.org/directory'
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
      - http01:
          ingress:
            class: nginx

Step-2: Before deploying my application to AWS EKS, I will configure a few more settings. I will create an IAM OIDC provider for the cluster and an Amazon EBS CSI driver IAM role for service accounts.

# Creating an IAM OIDC provider for your cluster
oidc_id=$(aws eks describe-cluster --name EKS1 --query "cluster.identity.oidc.issuer" --output text | cut -d '/' -f 5)
aws iam list-open-id-connect-providers | grep $oidc_id
eksctl utils associate-iam-oidc-provider --region=us-east-1 --cluster=EKS1 --approve

# check again
aws iam list-open-id-connect-providers | grep $oidc_id
# output
- Arn: arn:aws:iam::xxxxxxxxxx:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/xxxxxxxxxxxxxxxxxxxxxxx

Step-3: To create your Amazon EBS CSI plugin IAM role with eksctl, use the eksctl create iamserviceaccount command to provision the necessary IAM role and attach it to your EKS cluster.

eksctl create iamserviceaccount \
  --name ebs-csi-controller-sa \
  --namespace kube-system \
  --cluster EKS1 \
  --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
  --approve \
  --role-only \
  --role-name AmazonEKS_EBS_CSI_DriverRole

Step-4: Now, I will deploy the sample application and ingress-nginx.

# ingress-nginx.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: id-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
    service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - id.mehmetkanus
    secretName: tls-app
  rules:
    - host: id.mehmetkanus.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: web-service
                port:
                  number: 80
    - host: id.mehmetkanus.com
      http:
        paths:
          - path: /result
            pathType: Prefix
            backend:
              service:
                name: result-service
                port:
                  number: 80

Step-5: Now, let’s create an alias A record in Route 53 with the DNS name of our application’s load balancer and view the entire interface of our site.

Install Velero to EKS1:

1- Create an S3 Bucket to store backups.

# Replace <BUCKETNAME> and <REGION> with your own values below.

BUCKET=<BUCKETNAME>
REGION=<REGION>
aws s3 mb s3://$BUCKET --region $REGION

2- create IAM Policy.

cat > velero_policy.json <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeVolumes",
                "ec2:DescribeSnapshots",
                "ec2:CreateTags",
                "ec2:CreateVolume",
                "ec2:CreateSnapshot",
                "ec2:DeleteSnapshot"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:DeleteObject",
                "s3:PutObject",
                "s3:AbortMultipartUpload",
                "s3:ListMultipartUploadParts"
            ],
            "Resource": [
                "arn:aws:s3:::${BUCKET}/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::${BUCKET}"
            ]
        }
    ]
}
EOF

aws iam create-policy \
    --policy-name VeleroAccessPolicy \
    --policy-document file://velero_policy.json

3- Create Service Accounts for Velero.

PRIMARY_CLUSTER=EKS1
ACCOUNT=$(aws sts get-caller-identity --query Account --output text)

eksctl create iamserviceaccount \
    --cluster=$PRIMARY_CLUSTER \
    --name=velero-server \
    --namespace=velero \
    --role-name=eks-velero-backup \
    --role-only \
    --attach-policy-arn=arn:aws:iam::$ACCOUNT:policy/VeleroAccessPolicy \
    --approve

4- Install Velero in EKS1 Clusters.

# add helm repo
helm repo add vmware-tanzu https://vmware-tanzu.github.io/helm-charts

# create values.yaml file for install velero helm chart
cat > values.yaml <<EOF
configuration:
  backupStorageLocation:
  - bucket: $BUCKET
    provider: aws
  volumeSnapshotLocation:
  - config:
      region: $REGION
    provider: aws
initContainers:
- name: velero-plugin-for-aws
  image: velero/velero-plugin-for-aws:v1.7.1
  volumeMounts:
  - mountPath: /target
    name: plugins
credentials:
  useSecret: false
serviceAccount:
  server:
    annotations:
      eks.amazonaws.com/role-arn: "arn:aws:iam::${ACCOUNT}:role/eks-velero-backup"
EOF
 
# install velero to EKS1
helm install velero vmware-tanzu/velero \
    --create-namespace \
    --namespace velero \
    -f values.yaml

5- Backup all resources in all namespaces including persistent volumes.

Step-6: destroy EKS1

Step-7: create EKS2 for restore

• As seen, my newly created EKS2 Kubernetes cluster does not have any applications or Velero installed. Now, first and foremost, we will install Velero on EKS2 and then proceed to restore from the backup previously created on EKS1 to EKS2.

Step-8: For the EKS2 cluster, we will create an “IAM OIDC provider” and an “Amazon EBS CSI driver IAM role for service accounts.”

# Creating an IAM OIDC provider for your cluster
oidc_id=$(aws eks describe-cluster --name EKS2 --query "cluster.identity.oidc.issuer" --output text | cut -d '/' -f 5)
aws iam list-open-id-connect-providers | grep $oidc_id
eksctl utils associate-iam-oidc-provider --region=us-east-1 --cluster=EKS2 --approve

# check again
aws iam list-open-id-connect-providers | grep $oidc_id
# output
- Arn: arn:aws:iam::xxxxxxxxxx:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/xxxxxxxxxxxxxxxxxxxxxxx

Install Velero to EKS2:

• Previously, an S3 bucket was already set up for backups, and a policy named “VeleroAccessPolicy” was created for the service account. Now, for the EKS2 cluster, we will only create the service account and proceed to install Velero.

1- Create Service Accounts for Velero.

RECOVERY_CLUSTER=EKS2
ACCOUNT=$(aws sts get-caller-identity --query Account --output text)

eksctl create iamserviceaccount \
    --cluster=$RECOVERY_CLUSTER \
    --name=velero-server \
    --namespace=velero \
    --role-name=eks-velero-recovery \
    --role-only \
    --attach-policy-arn=arn:aws:iam::$ACCOUNT:policy/VeleroAccessPolicy \
    --approve

2- Install Velero in EKS2 Clusters.

# create values_recovery.yaml file for install velero helm chart
cat > values_recovery.yaml <<EOF
configuration:
  backupStorageLocation:
  - bucket: $BUCKET
    provider: aws
  volumeSnapshotLocation:
  - config:
      region: $REGION
    provider: aws
initContainers:
- name: velero-plugin-for-aws
  image: velero/velero-plugin-for-aws:v1.7.1
  volumeMounts:
  - mountPath: /target
    name: plugins
credentials:
  useSecret: false
serviceAccount:
  server:
    annotations:
      eks.amazonaws.com/role-arn: "arn:aws:iam::${ACCOUNT}:role/eks-velero-recovery"
EOF

# install velero to EKS2
helm install velero vmware-tanzu/velero \
    --create-namespace \
    --namespace velero \
    -f values_recovery.yaml

Step-9: Velero has been installed on EKS2, and as seen above, it can view the backups from EKS1 using the velero get backup command. Now, we will proceed to restore the backup from EKS1 to EKS2.

Step-10: If we update the previously created A record in the Route 53 service with the DNS name of the newly created load balancer, we will see all traffic directed to the same domain name.

Thank you for adding my article to your reading list! If you enjoyed it and found it helpful, please consider following me and giving the article a clap. Your support means a lot and helps me continue creating content that you love.

Thanks again, and happy reading!