Backup and Restore using Kasten’s K10 platform and OpenEBS

What is K10 ?

Kasten’s K10 platform is a unique application-centric data management solution for all public and private Kubernetes deployments that balances the needs of operators and developers making it easier to build, deploy, and manage stateful containerized applications at scale.

K10 provides the right balance and separation of concerns between developers and operators.

Main focus of K10 are:

• Backup and Recovery
• Application Migration
• Cluster cloning for Dev and test environments
• Disaster Recovery

Prerequisites

For this article we will be using two clusters

• baremetal cluster setup with RKE
• an AWS cluster setup with RKE
• AWS credentials for creating S3 bucket
• Helm
• Metrics server deployed on both clusters
• OpenEBS for persistent storage on baremetal cluster

You can setup cluster any which way you like or you can follow the instructions to setup a cluster on AWS using TK8’s cattle aws provisioner. Link is provided here.

Persistent storage is provided by OpenEBS as always, one of the best open-source container attached storage solutions currently.

Kasten Installation

Kasten installation requires a few prerequisites such as

• Your Kubernetes API server must have RBAC and the Aggregation Layer enabled.
• Helm configured with correct RBAC permissions

To make sure that you have everything setup and ready, run this preflight checks provided by Kasten

$ curl <https://docs.kasten.io/tools/k10_preflight.sh> | bash

If you don’t have helm installed with correct RBAC permissions (chiefly a service account that has cluster-admin rolebinding ). You can set it up with below commands

# add a service account within a namespace to segregate tiller
$ kubectl --namespace kube-system create sa tiller
# create a cluster role binding for tiller
$ kubectl create clusterrolebinding tiller \\
   --clusterrole cluster-admin \\
   --serviceaccount=kube-system:tiller
# initialize helm within the tiller service account
$ helm init --service-account tiller

Now that we have our preflight checks done, helm properly configured. Lets add the Kasten helm repository.

$ helm repo add kasten https://charts.kasten.io/

That’s it we are ready for our Kasten installation via helm. All we need is to configure out custom parameters that are specific to our cluster setup such as persistent storage used and the size of the persistent volumes you’d want etc. You can visit this link for a complete list of helm parameters you can modify according to your needs.

Since we’ve used OpenEBS as the persistent storage solution and its default storage class, our helm install command would be

# Modify the storageClass and pvc size according to the storage classes 
# present in your cluster.
$ helm install kasten/k10 --name=k10 --namespace=kasten-io \\
--set persistence.storageClass=openebs-jiva-default \\
--set persistence.size=20Gi

Installation on AWS

On AWS all you need to do is to provide AWS credentials to the helm command and also change the storage class to AWS default one (gp2)

helm install kasten/k10 --name=k10 --namespace=kasten-io \\
--set persistence.storageClass=gp2 \\
--set persistence.size=20Gi \\
--set secrets.awsAccessKeyId="${AWS_ACCESS_KEY_ID}" \\
--set secrets.awsSecretAccessKey="${AWS_SECRET_ACCESS_KEY}"

Dashboard Access

Once we have our Kasten deployments up, there are ways we can access the Kasten environment such as via a Dashboard, via kubectl (in the form of executing commands against Kasten CRDs). To know more about access via kubectl, you can find out more from this link https://docs.kasten.io/api/cli.html

In all honesty Dashboard access is the one thats most alluring as you can do all operations at the click of buttons giving users an ease of operation.

To get our dashboard access, lets do a port-forward first

$ kubectl --namespace kasten-io port-forward service/gateway 8080:8000

We can now access the K10 dashboard at http://127.0.0.1:8080/k10/#

Now that K10 is up, what can I do ?

Lets get into the operational side of things and onboard some of the features that K10 provides.

Application Protection

With K10 you can easily define policies to protect your applications, what that means is that you can provide define schedules for application snapshots and/or take manual snapshots when needed. Policies are used to automate your data management workflows.

Once you have configured a policy for an application, you can be rest assured that your application incase of any failure will have have its snapshots to fall back on.

Before we head onto to our Dashboard to configure policies, first lets setup wordpress on which we will configure the said policy.

To Deploy Wordpress, execute the below commands

# Create wordpress namespace
$ kubectl create namespace wordpress
# NOTE ::: This yaml only applies if your storage mechanism is OpenEBS
# For any other storage class, please download this yaml and 
# change the storage class name accordingly
$ kubectl apply -f <https://raw.githubusercontent.com/ipochi/k8s-bkp-restore/master/app/mysql-openebs.yaml>
$ kubectl apply -f <https://raw.githubusercontent.com/ipochi/k8s-bkp-restore/master/app/wordpress-openebs.yaml>

Now that we have our application setup we can proceed towards the dashboard.

Deployment changes for Kasten

We need to add a sidecar container with our wordpress and mysql containers to be able to extract data to be stored for backup. Nothing major just attaching a new sidecar container to both wordpress and mysql and mounting the same pvc to a different path.

- name: kanister-sidecar
        image: kanisterio/kanister-tools:0.20.0
        command: ["bash", "-c"]
        args:
        - "tail -f /dev/null"
        volumeMounts:
        - name: wordpress-persistent-storage
          mountPath: /data

####
- name: kanister-sidecar
        image: kanisterio/kanister-tools:0.20.0
        command: ["bash", "-c"]
        args:
        - "tail -f /dev/null"
        volumeMounts:
        - name: mysql-persistent-storage
          mountPath: /data2

Creating Kanister profile

Suppose we are using a storage provider that Kasten doesn’t provide [which we are in case OpenEBS], it doesn’t mean we are not covered. the application snapshots are then stored on S3 or S3-compatible object storage. There are slight modifications that need to be done to get our application snapshots and that begins by creating a Kanister profile. Once created we are good to go to take backups regardless of our underlying storage provider.

Head over to the Settings in the Dashboard home page, you’ll find the link to create a Kanister profile

Create a Kanister profile, providing AWS credentials and S3 bucket name. Please note the bucket name must exist prior to creating the Kanister profile, else the validation will fail.

Creating Policy

On the main page, you’ll notice the Applications box, click Unmanaged, you’ll notice our recently setup application Wordpress. Click on Create Policy.

What you’ll get is a plethora of options such as customizing the frequency of snapshots, retention policy, ability to export the snapshot for cross cluster migration. You can choose which resources to snapshot.

After you’ve created the policy in no time you’ll see a job running on the dashboard home page executing task as desired in the policy.

To confirm log in to the aws console and checkout the contents of the folder of the S3 bucket.

Restoring Applications

Restoring applications to a particular snapshot is a breeze. Just click on the application that you want to restore and click on Restore application.

Based on the previous backups, you'll be provided to choose a snapshot point. Sit back and watch your application gets restored.

Migrating Applications

This feature is absolutely important if you want to move applications across clusters whilst having your data present in the second cluster, or perhaps you want to clone a production environment in your test environment with same data as production to reproduce a critical bug that you've encountered.

Before we begin exporting our application, we need to create a Mobility Profile. We can create a mobility profile from settings page.

Select a profile for import and provide the AWS credentials for S3 storage and the name of the bucket.

We move on to exporting our application.To be able to restore application across cluster we need to export our application.

Similar to Restore application above , click on Export application and you'll be presented with the snapshots which you want to use for exporting the application, you'll get a token string, copy that, we will need in to our second cluster and we are done.

After a few minutes you'll notice on the dashboard that the export job has successfully finished.

To recover our application in another cluster we need to follow similar steps as creating a Mobility profile, Kanister profile and creating an import policy.

Workarounds for our use case

In our use case we have a barematal cluster and AWS cluster.

In our baremetal cluster we have used OpenEBS as the underlying persistent storage provider, which means we had to insert a sidecar kanister container as showed above to be able to extract data for backup and push it to object storage in S3.

We also had a storage class named openebs-jiva-default, this storage class isn't present in our AWS cluster as we are not using OpenEBS there. Default class on AWS cluster is gp2.

To be able to successfully restore our application with data in AWS cluster we need to make the following changes.

• Create a storage class named openebs-jiva-default

# Save this file as openebs-gp2.yaml 
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata: 
  name: openebs-jiva-default
provisioner: kubernetes.io/aws-ebs
parameters: 
  type: gp2
  fsType: ext4

Save the above file and execute the below command to create a Storage class

kubectl apply -f openebs-gp2.yaml

• Once the restored deployment is up, we will need to edit the wordpress service. In our baremetal cluster, we had used the NodePort 32555. Upon restore there is a possibility that a new port is assigned instead of 32555.
• Execute the below command and edit the NodePort to 32555/

kubectl edit -n wordpress svc wordpress

Upon doing this necessary changes, you’ll find that your application is up and working with the original data from your baremetal cluster

Conclusion

Kasten's K10 platform is great for click management of your application data. Ofcourse there are opensource solutions out there such as Heptio's Velero but what you get with K10 is seamless recovery management of your applications without being a kubectl ninja.

Plus if your cluster has three or less worker nodes you get all of the above said functionality and more absolutely free.