Backup and Restore Your Kubernetes resources and persistent volumes Using Velero
What is Velero?
Velero is an open source tool developed by VMware and used for backing and restoring resources in a Kubernetes cluster, performing disaster recovery, and can be used for migrating resources and persistent volumes to another Kubernetes cluster.
Velero features such as scheduled backups, retention schedules, and pre- or post-backup hooks for custom actions. Velero can help protect data stored in persistent volumes and makes your entire Kubernetes cluster more resilient.
In this article I have explained how to backup and Restore kubernates Resources and used Minio - a High Performance Object Storage which is API compatible with Amazon S3 cloud storage service.
Features of Velero:
- Back up Clusters: Backup your Kubernetes resources and volumes for an entire cluster, or part of a cluster by using namespaces or label selectors.
- Schedule Backups: Set schedules to automatically kickoff backups at recurring intervals.
- Backup Hooks: Configure pre and post-backup hooks to perform custom operations before and after Velero backups.
How Velero Works?
Each Velero operation like backup, restore is a custom resource defined with a Kubernetes Custom Resource Definition (CRD) and stored in etcd. Velero also includes controllers that process the custom resources to perform backups, restores, and all related operations.
When you run velero backup create backup1:
- The Velero client makes a call to the Kubernetes API server to create a Backup object.
- The Backup Controller notices the new Backup object and performs validation.
- The BackupController begins the backup process. It collects the data to back up by querying the API server for resources.
- The BackupController makes a call to the object storage service – for example, AWS S3 – to upload the backup file.
Installation:
Prerequisite:
- docker should be installed.
- Kubernetes cluster (v1.20)
What is Minio?
MiniO has an API which is compatible with Amazon S3. This will allow you to develop applications locally and later, easily port them to AWS.
Step 1: Minio Installation on docker
Use the following commands to run a standalone MiniO server as a container.
NOTE: To deploy MiniO on with persistent storage, you must map local persistent directories from the host OS to the container using example, -v /root:/data maps the host OS drive at /root to /data on the container.
root@devopsguyvm:~# docker run -p 9000:9000 -p 9001:9001 --name minio1 -v /root:/data minio/minio server /data --console-address ":9001"
API: http://172.17.0.2:9000 http://127.0.0.1:9000Console: http://172.17.0.2:9001 http://127.0.0.1:9001Documentation: https://docs.min.io
WARNING: Detected default credentials 'minioadmin:minioadmin', we recommend that you change these values with 'MINIO_ROOT_USER' and 'MINIO_ROOT_PASSWORD' environment variables
Once Command is successful, it will give you API and console URL. You can Open Minio console UI using below URL http://127.0.0.1:9001 on localhost.For first installation default username and password is minioadmin/minioadmin . You can find username/Password which is access_key and secret_key Property inside file /data/.minio.sys/config/config.json in container.
root@devopsguyvm:~# docker exec -it minio1 sh
sh-4.4# cd /data/.minio.sys/config/
sh-4.4# ls
config.json iam
sh-4.4# cat config.json
Once Successful login click on the Object Browser ->Create Bucket and enter a bucket name which we are using to store backup.
Step 2 : Download Latest Velero 1.6.2 Release
root@devopsguyvm:~# wget https://github.com/heptio/velero/releases/download/v1.0.0/velero-v1.6.2-linux-amd64.tar.gz
root@devopsguyvm:~# tar zxf velero-v1.6.2-linux-amd64.tar.gz
root@devopsguyvm:~# sudo mv velero-v1.6.2-linux-amd64/velero /usr/local/bin/You can validate the if Velero CLI is installed successfully using below commands.
root@devopsguyvm:~# which velero
/usr/local/bin/velero
root@devopsguyvm:~# velero version
Client:
Version: v1.6.2
Git commit: 8c9cdb9603446760452979dc77f93b17054ea1cc
Server:
root@devopsguyvm:~# velero
Velero is a tool for managing disaster recovery, specifically for Kubernetes
cluster resources. It provides a simple, configurable, and operationally robust
way to back up your application state and associated data.If you're familiar with kubectl, Velero supports a similar model, allowing you to
execute commands such as 'velero get backup' and 'velero create schedule'. The same
operations can also be performed as 'velero backup get' and 'velero schedule create'.Usage:
velero [command]Available Commands:
backup Work with backups
Step 3: Create credentials file for Velero initialization
Velero need minio.credentials file for to talk with Object store (May be Amazon s3, minio in our case) using which Velero will Authenticate with minio and store the backups.
root@devopsguyvm:~# cat minio.credentials
[default]
aws_access_key_id=minioadmin
aws_secret_access_key=minioadminStep4: Install Velero in the Kubernetes Cluster
Once You enter below command, Velero will check the kubeconfig file and use that Cluster for Velero deployment. It will first Create the velero namespace inside Kubernates cluster and deploy velero as deployment and many Custom resources also created.
Please note : we are using provider as aws because minio object Storage is compatible with Amazon S3.
root@devopsguyvm:~# velero install --plugins velero/velero-plugin-for-aws:v1.0.0 --provider aws --bucket demo --secret-file ./minio.credentials --backup-location-config region=minio,s3ForcePathStyle=true,s3Url=http://127.0.0.1:9000Check All the resources from velero namespace.
root@devopsguyvm:~# kubectl get pod -n velero
NAME READY STATUS RESTARTS AGE
velero-57c675458b-ssnsp 1/1 Running 0 21m
root@devopsguyvm:~# kubectl get all -n velero
NAME READY STATUS RESTARTS AGE
pod/velero-57c675458b-ssnsp 1/1 Running 0 21mNAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/velero 1/1 1 1 21mNAME DESIRED CURRENT READY AGE
replicaset.apps/velero-57c675458b 1 1 1 21m
Check all the CRDs created as a part of Velero Installation.
root@devopsguyvm:~# kubectl get crd -n velero
NAME CREATED AT
bgpconfigurations.crd.projectcalico.org 2021-07-29T10:41:51Z
bgppeers.crd.projectcalico.org 2021-07-29T10:41:52Z
blockaffinities.crd.projectcalico.org 2021-07-29T10:41:52Z
felixconfigurations.crd.projectcalico.org 2021-07-29T10:41:52Z
clusterinformations.crd.projectcalico.org 2021-07-29T10:41:52Z
globalnetworkpolicies.crd.projectcalico.org 2021-07-29T10:41:52Z
globalnetworksets.crd.projectcalico.org 2021-07-29T10:41:53Z
hostendpoints.crd.projectcalico.org 2021-07-29T10:41:54Z
ipamblocks.crd.projectcalico.org 2021-07-29T10:41:55Z
ipamconfigs.crd.projectcalico.org 2021-07-29T10:41:55Z
ipamhandles.crd.projectcalico.org 2021-07-29T10:41:56Z
ippools.crd.projectcalico.org 2021-07-29T10:41:57Z
networkpolicies.crd.projectcalico.org 2021-07-29T10:41:58Z
networksets.crd.projectcalico.org 2021-07-29T10:41:58Z
volumesnapshotlocations.velero.io 2021-07-29T10:44:11Z
backups.velero.io 2021-07-29T10:44:11Z
restores.velero.io 2021-07-29T10:44:11Z
podvolumebackups.velero.io 2021-07-29T10:44:11Z
resticrepositories.velero.io 2021-07-29T10:44:11Z
schedules.velero.io 2021-07-29T10:44:12Z
backupstoragelocations.velero.io 2021-07-29T10:44:12Z
downloadrequests.velero.io 2021-07-29T10:44:12Z
deletebackuprequests.velero.io 2021-07-29T10:44:13Z
podvolumerestores.velero.io 2021-07-29T10:44:13Z
serverstatusrequests.velero.io 2021-07-29T10:44:13ZNow Velero is Successfully Installed. Lets try to create a demo namespace, deploy nginx application inside demo namespace and Lets try backup and restore it using Velero.
Step5: Backup and Restore Resources Using Velero.
You can List out Provider name,bucket name and Access mode etc using below command. Velero will store all your backups inside below Bucket.
root@devopsguyvm:~# velero backup-location get
NAME PROVIDER BUCKET/PREFIX PHASE LAST VALIDATED ACCESS MODE DEFAULT
default aws demo Unknown Unknown ReadWrite trueLets create a demo namespace and deploy nginx Application inside it.
root@devopsguyvm:~# kubectl create ns demo
root@devopsguyvm:~# kubectl create deploy nginx --image=nginx -n demo
deployment.apps/nginx created
root@devopsguyvm:~# kubectl get all -n demo
NAME READY STATUS RESTARTS AGE
pod/nginx-84784697bb-8k5bs 1/1 Running 0 2m55sNAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/nginx 1/1 1 1 32mNAME DESIRED CURRENT READY AGE
replicaset.apps/nginx-84784697bb 1 1 1 2m55s
replicaset.apps/nginx-865fffb4b4 0 0 0 32m
Please note: My first backup was failed with the error (not able to to connec-t localhost and 9000 port). Then I have changed s3Url with container URL instead of Localhost inside backupstoragelocation custom resource.
kubectl -n velero edit backupstoragelocation defaults3Url: http://172.17.0.2:9000
Lets backup all the resources from demo namespace with below command.You can backup all the namespace or you can take backup from specific namespace as well. You can include/exclude specific resource using — -include-resources /--exclude-resources . (e.g. to backup pods from demo namespace velero backup create backup3 — include-namespaces demo --include-resources pods).
oot@devopsguyvm:~# velero backup create backup3 --include-namespaces demo
Backup request "backup3" submitted successfully.
Run `velero backup describe backup3` or `velero backup logs backup3` for more details.root@devopsguyvm:~# velero backup describe backup3
Name: backup3
Namespace: velero
Labels: velero.io/storage-location=default
Annotations: velero.io/source-cluster-k8s-gitversion=v1.20.9-34+23bd09b0412ac4
velero.io/source-cluster-k8s-major-version=1
velero.io/source-cluster-k8s-minor-version=20+Phase: Completed
List all the backups using below command.
root@devopsguyvm:~# velero get backup
NAME STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION SELECTOR
backup1 Failed 0 0 2021-07-29 17:05:02 +0530 IST 29d default <none>
backup2 Failed 0 0 2021-07-29 17:24:25 +0530 IST 29d default <none>
backup3 Completed 0 0 2021-07-29 17:46:44 +0530 IST 29d default <none>
firstbackup Failed 0 0 2021-07-29 17:04:02 +0530 IST 29d default <none>You can now see backup created successfully in Minio console UI.
Now Lets delete the demo namespace and restore it from the backup3.
root@devopsguyvm:~# kubectl delete ns demo
namespace "demo" deletedIf Previously if you restore any backup the it can be view using velero restore get command. Use below command to restore demo namespace from backup3.
root@devopsguyvm:~# velero restore get
root@devopsguyvm:~# velero restore create restore1 --from-backup backup3
Restore request "restore1" submitted successfully.
Run `velero restore describe restore1` or `velero restore logs restore1` for more details.Check if restoration is Successful or not with the below command.
root@devopsguyvm:~# kubectl get all -n demo
NAME READY STATUS RESTARTS AGE
pod/nginx-84784697bb-gjwvw 1/1 Running 0 17sNAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/nginx 1/1 1 1 108sNAME DESIRED CURRENT READY AGE
replicaset.apps/nginx-84784697bb 1 1 1 17s
replicaset.apps/nginx-865fffb4b4 0 0 0 108sroot@devopsguyvm:~# velero restore get
NAME BACKUP STATUS STARTED COMPLETED ERRORS WARNINGS CREATED SELECTOR
restore1 backup3 Completed 2021-07-29 18:00:43 +0530 IST 2021-07-29 18:00:44 +0530 IST 0 0 2021-07-29 18:00:4
Step6: Scheduled backup
You can set Velero to take scheduled backups as well. Use below command for it.
# Create a backup every 6 hours
velero schedule create backupschedule --schedule="0 */6 * * *"
# Create a backup every 6 hours with the @every notation
velero schedule create backupschedule --schedule="@every 6h"
# Create a daily backup of the web namespace
velero schedule create backupschedule --schedule="@every 24h" --include-namespaces web
# Create a weekly backup, each living for 90 days (2160 hours)
velero schedule create backupschedule --schedule="@every 168h" --ttl 2160h0m0sVelero also used extensively to migrate and restore Kubernates clusters. Migrating one cluster to another cluster is slightly more complicated as you have to bootstrap the minimum resources necessary and install velero correctly to perform a restore.
