Backup and Restore of Contrail Service Orchestration (CSO) Databases
This document introduces the backup and restore capabilities available in Contrail Service Orchestration (CSO). It provides an overview of the concepts, command options, and some examples of how to manage these functions. Although CSO is a GUI-based application, the backup and restore operations can only be managed from the CLI of the startup virtual machine (startup-vm). See the Operations for details. XYZ
CSO Database Backup and Restore
The Contrail Service Orchestration (CSO) architecture is made up of several virtual machines, each handling pieces of the workload required to make CSO function. These virtual machines store and access their working data in various databases. In order for CSO to function properly, all of the running databases must be functioning properly. Backup and restore of this critical data is key to ensuring that your CSO installation is running at its best. Starting in CSO 4.1, full backup of all platform, op-co, tenant, and customer data can be run manually or periodically and that data can be restored from the backups when and if the need arises.
Figure 1 shows a conceptual image of how backup and restore is implemented in CSO 4.1. The database systems that are currently backed up within the framework are: MariaDB, Cassandra, ElasticSearch, ArangoDB, Zookeeper, ETCD, Icinga, Swift, and K8ETCD. You can backup and restore data for Contrail Analytics. The system also backs up encrypted passwords, and system certificates so that restoring data from any specific backup puts CSO back into the state it was in at the time of that backup.
You can selectively
restore ETCD keys. Before you restore a specific ETCD key, you must
add the specific ETCD key in the
"etcd":["/saml", "/routingmanager/sdwan/asnumber", "/natregions/regional", "/csp/infra/swift/naturl"]
Any changes made between the last backup and the current restoration are lost. Generally, backups are made on a system-wide basis meaning that individual op-co or tenant data can not be backed up or restored apart from the rest of the system data.
While it is possible to backup and restore individual databases, there are risks when doing this since the restored database might not be able to fully synch with the current states of the existing databases. This is especially true if there is a long period of time between the backup and restore operations.
The backup and restore operations work with high-availability (HA). This document describes the configuration, scheduling, and operation of backup and restore procedures in CSO.
Backup and restore are critical tasks that touch every data storage system used by CSO. Juniper relieves you of the burden of configuring backup details by automatically setting up everything needed to backup and restore CSO during the installation process. No configuration is needed.
Although there is no major interaction between the user and the underlying components that make up the backup framework, it is helpful to know the functions that each of the components perform. Table 1 lists the major components and a brief description of each.
Table 1: Major Components
Backup and Restore Controller
All of the backup and restore operations are performed using the command line interface (CLI) of the startup-vm machine. The user in charge of the operations logs onto the startup-vm over ssh and performs any needed operations. Figure 2 shows the flow of backup and restore operations.
For backup operations, run the cso_backupnrestore command on the startup-vm, using the proper arguments for backing up an individual database or all of the databases. When this happens, the backup and restore controller communicates the backup request to the individual plug-ins using the SaltStack message bus. The plug-ins that reside on the various central and regional vms receive the message and trigger the needed action.
Backups are stored in the /backups/ directory on the startup-vm. This location can not be changed. The storage for this location can be local to the startup-vm or it can be located on a Storage Area Network (SAN).
For restore operations, the same cso_backupnrestore command is used with different options as described in Table 2 below. System stability is confirmed, and the needed restore commands are sent to the plug-ins for each database as needed. Once the restore is finished, CSO checks for system stability again, does any required cleanup and puts itself back into operational mode.
The CLI command used to create backups or restore files from backup is named cso_backupnrestore.
Options available for the cso_backupnrestore command are shown in Table 2. Only one of the arguments can be used with any one of the options.
Table 2: cso_backupnrestore Command Options
Specify operation (REQUIRED)
backup, restore, healthcheck, reindex, backupdetails, listbackups, scheduledbackup
Specify the name of the snapshot created by backup operation or restored by restore operation.
Specify which database to backup or restore [default ‘*’](OPTIONAL)
For backup: only ‘*’ is allowed.
For restore: Comma separated list with any or all of: cassandra, elasticsearch, zookeeper, mariadb, etcd, arrangodb. ’*’ restores all databases
Specify whether the restore operation is for disaster recovery or not [Default no].
yes or no
Set cron job parameters for backup operation.
Only valid in combination with schedulebackup argument for the -b option.
m-h-dom-mon-dow-m [-m yes]
Backup and Restore Examples
IP address of the startup virtual machine (startup-vm) of your CSO instance
Root access to the startup-vm using the ssh protocol
The following commands must be run at the command line interface of the startup-vm of CSO. The location and access credentials needed to access the startup-vm in your CSO installation should be known to you or the person or group who installed CSO.
This example performs a simple backup of all CSO databases
into the directory
cso_backupnrestore -b backup -s MAR09
Scheduled Backup Using Cron-job
This example creates a scheduled backup that runs in
maintenance mode every Sunday afternoon at 1:00 PM and stores the
backup in the
/bakups/DAILY/<timestamp>/ directory. The timestamp directory is created when the backup starts.
cso_backupnrestore -b scheduledbackup -z 0-13-*-*-0
This example restores the backup located in the
cso_backupnrestore -b restore -s /backups/DAILY-09/2019-03-16T04 -r no
This example performs a disaster recovery restore operation
from the backup located in the
cso_backupnrestore -b restore -s /backups/DAILY-09/2019-03-16T04 -r yes
Health Check Example
This example performs a health check on the CSO installation.
cso_backupnrestore -b healthcheck
This example performs a reindex of the Elasticsearch database.
cso_backupnrestore -b reindex
Restore a Specific Component Example
This example restores a specific component, ETCD.
cso_backupnrestore -b restore -s /backups/DAILY-09/2018-03-16T04 -c ‘etcd’
Running Health heck for a Specific Component Example
This example performs the health check of a specific component, ETCD.
cso_backupnrestore -b healthcheck -c etcd
Clear Index for Elasticsearch Example
This example clears the index for Elasticsearch.
cso_backupnrestore -b clear_indices
Repair Nodetool in Cassandra Example
This example repairs the Nodetool in Cassandra.
cso_backupnrestore -b nodetool_repair
List Available Backups Example
This example lists all backups that are available.
cso_backupnrestore -b listbackups
View Details of a Specific Backup Example
This example lists the details of a specific backup.
cso_backupnrestore -b backupdetails -s MAR09