Back Up and Restore Contrail Etcd

SUMMARY Learn how to back up and restore the Contrail etcd database.

CN2 stores data in its own etcd database. Use the procedures in this section to back up and restore the Contrail etcd database.

Back Up the Contrail Etcd Database

Use this example procedure to back up the Contrail etcd database.

Install etcdctl on all control plane nodes.

Download etcd. This example downloads to the /tmp directory.

Untar and move the etcd executable to a directory in your path (for example /usr/local/bin).

Check that you've installed etcd.

Get a list of the contrail-etcd pods.
Take note of the contrail-etcd pod names, the IP addresses, and the nodes they're running on. You'll need this information in the next few steps.
Copy the etcd certificate and key files from the pods to the control plane nodes.
We run kubectl on the control plane nodes in this step. We assume you've set up kubeconfig on the control plane nodes in its default location (~/.kube/config).
1. Pick a contrail-etcd pod (for example, contrail-etcd-0) and log in to the control plane node that's hosting that pod.
2. Copy the certificate and key files from that contrail-etcd pod to the hosting control plane node.
  In this example, we're copying the certificates and key files from the contrail-etcd-0 pod to local files on this node.
  This copies the certificate and key files from the contrail-etcd-0 pod to ca.crt, tls.crt, and tls.key in the current directory on this control plane node.
3. Repeat for each contrail-etcd pod.
Back up the etcd database on one of the control plane nodes. You only need to back up the database on one node.
1. Log back in to one of the control plane nodes.
2. Back up the etcd database.
  This example saves the database to /tmp/etcdbackup.db on this control plane node.
  where <etcd-pod-ip> is the IP address of the pod on this node and the <etcd-port> is the port that etcd is listening on (by default, 12379).
Copy the database to a safe location.

Use this example procedure to restore the Contrail etcd database from a snapshot.

Copy the snapshot you want to restore to all the control plane nodes.
The steps below assume you've copied the snapshot to /tmp/etcdbackup.db on all the control plane nodes.
Restore the snapshot.
1. Log in to one of the control plane nodes. In this example, we're logging in to the control plane node that is hosting contrail-etcd-0.
2. Restore the etcd database to the contrail-etcd-0 pod on this control plane node.
  This creates a contrail-etcd-0.etcd directory on the node.
  where --name=contrail-etcd-0 specifies that this command is restoring the database to contrail-etcd-0, --initial-cluster=... lists all the contrail-etcd members in the cluster, and --initial-advertise-peer-urls=... refers to the IP address and port number that the contrail-etcd-0 pod is listening on.
3. Repeat for the other contrail-etcd pods on their respective control plane nodes, substituting the --name and --initial-advertise-peer-urls values with the respective contrail-etcd pod name and IP address.

Stop the contrail-etcd pods.

This sets the replicas to 0, which effectively stops the pods.

Replace contrail-etcd data with the data from the snapshot.
1. SSH into one of the control plane nodes.
2. Replace the data. Recall that the snapshot is stored in the contrail-etcd-<xxx>.etcd directory.
  where contrail-etcd-xxx is the name of the contrail-etcd pod on the control plane node that you logged in to.
3. Repeat for the other control plane nodes.

Start the contrail-etcd pods.

This sets the replicas to 3, which effectively starts the pods.

Restart the contrail-system apiserver and controller.
Delete all the contrail-k8s-apiserver and contrail-k8s-controller pods.
These pods will automatically restart.
Restart the vrouters.
Delete all the contrail-vrouter-masters and contrail-vrouter-nodes pods.
These pods will automatically restart.

Check that all pods are in running state.