# Backup Kubernetes Cluster using Velero

> Canonical: https://zufardhiyaulhaq.com/backup-kubernetes-cluster-using-velero/
> Tags: [kubernetes, tooling]

Velero gives you tools to back up and restore your Kubernetes cluster resources and persistent volumes. Velero have ability to backup cluster, migrate cluster resource to other cluster, and also replicate cluster to other cluster.

This is my first time trying Velero. correct me if I am wrong.

## Requirement
- docker CLI
- docker-compose CLI
- velero CLI
- minio (mc) CLI
- virtualbox
- minikube

### Setup Minio
Minio running as storage backend for velero to store the backup, you can using other storage backend like AWS S3, GCP GCS, and other defined in [backend provider documentation](https://velero.io/docs/v1.4/supported-providers/). For simplicity, we use Minio.

- Install Minio using docker-compose
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">version: <span class="s1">'3.7'</span>

services:
  minio:
    container_name: minio-storage
    image: minio/minio:RELEASE.2020-05-29T14-08-49Z
    volumes:
      - minio-storage:/minio-storage
    ports:
      - <span class="s2">"9000:9000"</span>
    environment:
      MINIO_ACCESS_KEY: zufar_minio_access_key
      MINIO_SECRET_KEY: zufar_minio_secret_key
    <span class="nb">command</span>: server /minio-storage
    healthcheck:
      <span class="nb">test</span>: <span class="o">[</span><span class="s2">"CMD"</span>, <span class="s2">"curl"</span>, <span class="s2">"-f"</span>, <span class="s2">"http://localhost:9000/minio/health/live"</span><span class="o">]</span>
      interval: 30s
      <span class="nb">timeout</span>: 20s
      retries: 3

volumes:
  minio-storage:</code></pre></figure>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell">docker-compose up <span class="nt">-d</span></code></pre></figure>

- get minio IP
<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nb">export </span><span class="nv">MINIO_IP</span><span class="o">=</span><span class="si">$(</span>docker inspect <span class="nt">-f</span> <span class="s1">'{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}'</span> minio-storage<span class="si">)</span>
<span class="nb">echo</span> <span class="s2">"Minio IP: </span><span class="nv">$MINIO_IP</span><span class="s2">"</span></code></pre></figure>

- create storage bucket

You can create bucket from web dashboard or via CLI.
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">mc config host add minio-storage http://<span class="nv">$MINIO_IP</span>:9000 zufar_minio_access_key zufar_minio_secret_key
mc mb minio-storage/velero-bucket
mc policy <span class="nb">set </span>public minio-storage/velero-bucket</code></pre></figure>

### Setup Kubernetes

For simplicity, we using minikube to run Kubernetes cluster.
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">minikube start <span class="nt">--driver</span><span class="o">=</span>virtualbox <span class="nt">--force</span><span class="o">=</span><span class="nb">true</span> <span class="nt">--kubernetes-version</span><span class="o">=</span><span class="s1">'1.15.11'</span> <span class="nt">--memory</span> 16128 <span class="nt">--cpus</span> 8 <span class="nt">--profile</span> cluster-velero</code></pre></figure>

### Setup Velero
Velero have two system, the client-side and server-side. First install the CLI (client-side) using brew or asdf or other. It is depended on your environment.

Velero installation is a little complex, for now lets just use `velero install`. You must run this command from environment that have access directly to Kubernetes cluster (have kubectl working).

- Add Velero credential
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">vi credentials-velero

<span class="o">[</span>default]
aws_access_key_id <span class="o">=</span> zufar_minio_access_key
aws_secret_access_key <span class="o">=</span> zufar_minio_secret_key</code></pre></figure>

- Install Velero in Kubernetes
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">velero <span class="nb">install</span> <span class="se">\</span>
    <span class="nt">--provider</span> aws <span class="se">\</span>
    <span class="nt">--plugins</span> velero/velero-plugin-for-aws:v1.0.0 <span class="se">\</span>
    <span class="nt">--bucket</span> velero-bucket <span class="se">\</span>
    <span class="nt">--secret-file</span> ./credentials-velero <span class="se">\</span>
    <span class="nt">--backup-location-config</span> <span class="nv">region</span><span class="o">=</span>minio,s3ForcePathStyle<span class="o">=</span><span class="s2">"true"</span>,s3Url<span class="o">=</span>http://<span class="nv">$MINIO_IP</span>:9000 <span class="se">\</span>
    <span class="nt">--snapshot-location-config</span> <span class="nv">region</span><span class="o">=</span>minio,s3ForcePathStyle<span class="o">=</span><span class="s2">"true"</span>,s3Url<span class="o">=</span>http://<span class="nv">$MINIO_IP</span>:9000</code></pre></figure>

- Make sure velero working
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">kubectl <span class="nt">-n</span> velero get pod
kubectl <span class="nt">-n</span> velero get VolumeSnapshotLocation
kubectl <span class="nt">-n</span> velero describe VolumeSnapshotLocation
kubectl <span class="nt">-n</span> velero get BackupStorageLocation
kubectl <span class="nt">-n</span> velero describe BackupStorageLocation</code></pre></figure>

## Simulate disaster

### Without PVC
- create namespace & deployment
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">kubectl apply <span class="nt">-f</span> https://gist.githubusercontent.com/zufardhiyaulhaq/c7d59b0725e6c23e5bf6daeeb82b3333/raw/778986f376c5aeef59ec528a1f3a510e0baaab7f/deployment-svc-kubernetes

kubectl <span class="nt">-n</span> nginx-example get pod
kubectl <span class="nt">-n</span> nginx-example get svc</code></pre></figure>

- create namespace backup
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">velero backup create nginx-backup <span class="nt">--include-namespaces</span> nginx-example</code></pre></figure>

- check backup progress
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">velero backup describe nginx-backup
velero backup logs nginx-backup</code></pre></figure>
You can also check Minio bucket.

- simulate disaster

Please don't delete namespace in production cluster guys
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">kubectl delete namespace nginx-example</code></pre></figure>

- Check namespace & deployment already deleted
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">kubectl get namespaces
kubectl <span class="nt">-n</span> nginx-example get pod
kubectl <span class="nt">-n</span> nginx-example get svc</code></pre></figure>

- Restore namespaces
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">velero restore create <span class="nt">--from-backup</span> nginx-backup</code></pre></figure>

- Check namespace & deployment already restored
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">kubectl get namespaces
kubectl <span class="nt">-n</span> nginx-example get pod
kubectl <span class="nt">-n</span> nginx-example get svc</code></pre></figure>

### With PVC
- delete previous namespace
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">kubectl delete namespace nginx-example</code></pre></figure>

- create namespace & deployment
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">kubectl apply <span class="nt">-f</span> https://gist.githubusercontent.com/zufardhiyaulhaq/2ce21ca351807991d8a370315695faee/raw/4c0575f2c74aa26fe32f4eff555be3efcf15e23c/deployment-svc-kubernetes-pvc

kubectl <span class="nt">-n</span> nginx-example get pod
kubectl <span class="nt">-n</span> nginx-example get svc
kubectl <span class="nt">-n</span> nginx-example get pvc</code></pre></figure>

- create namespace backup
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">velero backup create nginx-backup-pvc <span class="nt">--include-namespaces</span> nginx-example</code></pre></figure>

- check backup progress
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">velero backup describe nginx-backup-pvc
velero backup logs nginx-backup-pvc</code></pre></figure>
You can also check Minio bucket.

- simulate disaster

Please don't delete namespace in production cluster guys
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">kubectl delete namespace nginx-example</code></pre></figure>

- Check namespace & deployment already deleted
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">kubectl get namespaces
kubectl <span class="nt">-n</span> nginx-example get pod
kubectl <span class="nt">-n</span> nginx-example get svc
kubectl <span class="nt">-n</span> nginx-example get pvc</code></pre></figure>

- Restore namespaces
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">velero restore create <span class="nt">--from-backup</span> nginx-backup-pvc</code></pre></figure>

- Check namespace & deployment already restored
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">kubectl get namespaces
kubectl <span class="nt">-n</span> nginx-example get pod
kubectl <span class="nt">-n</span> nginx-example get svc
kubectl <span class="nt">-n</span> nginx-example get pvc</code></pre></figure>

## Conclusion

Velero is a powerful tools to prevent disaster in Kubernetes also have many storage backend support and easy to recover. But before deploying to production as disaster recovery system, we must deep dive how Velero can backup PVC if you have stateful application. Because PVC have multiple backend.

