Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

Backup and Restore of Contrail Service Orchestration (CSO) Databases


This document introduces the backup and restore capabilities available in Contrail Service Orchestration (CSO). It provides an overview of the concepts, command options, and some examples of how to manage these functions. Although CSO is a GUI-based application, the backup and restore operations can only be managed from the CLI of the startup virtual machine (startup-vm). See the Operations for details. XYZ

CSO Database Backup and Restore

The Contrail Service Orchestration (CSO) architecture is made up of several virtual machines, each handling pieces of the workload required to make CSO function. These virtual machines store and access their working data in various databases. In order for CSO to function properly, all of the running databases must be functioning properly. Backup and restore of this critical data is key to ensuring that your CSO installation is running at its best. Starting in CSO 4.1, full backup of all platform, op-co, tenant, and customer data can be run manually or periodically and that data can be restored from the backups when and if the need arises.

Figure 1: Backup and Restore Concept
Backup and Restore

Figure 1 shows a conceptual image of how backup and restore is implemented in CSO 4.1. The database systems that are currently backed up within the framework are: MariaDB, Cassandra, ElasticSearch, ArangoDB, Zookeeper, ETCD, Icinga, Swift, and K8ETCD. You can backup and restore data for Contrail Analytics. The system also backs up encrypted passwords, and system certificates so that restoring data from any specific backup puts CSO back into the state it was in at the time of that backup.

You can selectively restore ETCD keys. Before you restore a specific ETCD key, you must add the specific ETCD key in the /usr/local/etc/etcd_keys.json file.

For example,

Any changes made between the last backup and the current restoration are lost. Generally, backups are made on a system-wide basis meaning that individual op-co or tenant data cannot be backed up or restored apart from the rest of the system data.


While it is possible to backup and restore individual databases, there are risks when doing this since the restored database might not be able to fully synch with the current states of the existing databases. This is especially true if there is a long period of time between the backup and restore operations.

The backup and restore operations work with high-availability (HA). This document describes the configuration, scheduling, and operation of backup and restore procedures in CSO.


Backup and restore are critical tasks that touch every data storage system used by CSO. Juniper relieves you of the burden of configuring backup details by automatically setting up everything needed to backup and restore CSO during the installation process. No configuration is needed.

Major Components

Although there is no major interaction between the user and the underlying components that make up the backup framework, it is helpful to know the functions that each of the components perform. Table 1 lists the major components and a brief description of each.

Table 1: Major Components



Backup and Restore Controller

  • Handles backup or restore calls from administrator. The calls are made using the cso_backupnrestore script that resides only on the startup-vm.

  • Communicates and delegates requests to individual plug-ins.

  • Manages lifecycle of backup and restore operations: pre-hook, backup and restore, and post-hook.

  • Salt Master

Plug-in Framework

  • Framework that allows backup and restore to deal with multiple different databases.

  • Allows for future inclusion of other databases.

  • Salt Minions


  • Addition of new plug-in has to adhere to standards.

  • All plug-ins are triggered by backup and restore controller.

  • Pre-hook, post hook and backup or restore operations are implemented by individual plug-ins.


All of the backup and restore operations are performed using the command line interface (CLI) of the startup-vm machine. The user in charge of the operations logs onto the startup-vm over ssh and performs any needed operations. Figure 2 shows the flow of backup and restore operations.

Figure 2: Backup and Restore Operations
Backup and Restore Operations

For backup operations, run the cso_backupnrestore command on the startup-vm, using the proper arguments for backing up an individual database or all of the databases. When this happens, the backup and restore controller communicates the backup request to the individual plug-ins using the SaltStack message bus. The plug-ins that reside on the various central and regional vms receive the message and trigger the needed action.

Backups are stored in the /backups/ directory on the startup-vm. This location cannot be changed.

For restore operations, the same cso_backupnrestore command is used with different options as described in Table 2 below. System stability is confirmed, and the needed restore commands are sent to the plug-ins for each database as needed. Once the restore is finished, CSO checks for system stability again, does any required cleanup and puts itself back into operational mode.

Command Usage

The CLI command used to create backups or restore files from backup is named cso_backupnrestore.

Options available for the cso_backupnrestore command are shown in Table 2. Only one of the arguments can be used with any one of the options.

Table 2: cso_backupnrestore Command Options





Specify operation (REQUIRED)

backup, restore, healthcheck, reindex, backupdetails, listbackups, scheduledbackup


Specify the name of the snapshot created by backup operation or restored by restore operation.

backup name


Specify which database to backup or restore [default ‘*’](OPTIONAL)

For backup: only ‘*’ is allowed.

For restore: Comma separated list with any or all of: cassandra, elasticsearch, zookeeper, mariadb, etcd, arrangodb. ’*’ restores all databases


Specify whether the restore operation is for disaster recovery or not [Default no].

yes or no


Set cron job parameters for backup operation.

Only valid in combination with schedulebackup argument for the -b option.

m-h-dom-mon-dow-m [-m yes]

  • m–minute (0-59)

  • h–hour (0-23)

  • dom–day of month (1-31)

  • mon–month (1-12)

  • dow–day of week (0-6)

Backup and Restore Examples


  • IP address of the startup virtual machine (startup-vm) of your CSO instance

  • Root access to the startup-vm using the ssh protocol

The following commands must be run at the command line interface of the startup-vm of CSO. The location and access credentials needed to access the startup-vm in your CSO installation should be known to you or the person or group who installed CSO.


This example performs a simple backup of all CSO databases into the directory /backup/MAR09/

Scheduled Backup Using Cron-job

This example creates a scheduled backup that runs in maintenance mode every Sunday afternoon at 1:00 PM and stores the backup in the /bakups/DAILY/<timestamp>/ directory. The timestamp directory is created when the backup starts.


This example restores the backup located in the /backups/DAILY-09/2019-03-16T04/ directory.

This example performs a disaster recovery restore operation from the backup located in the /backups/DAILY-09/2019-03-16T04/ directory.

Health Check Example

This example performs a health check on the CSO installation.

Reindex Example

This example performs a reindex of the Elasticsearch database.

Restore a Specific Component Example

This example restores a specific component, ETCD.

Running Health heck for a Specific Component Example

This example performs the health check of a specific component, ETCD.

Clear Index for Elasticsearch Example

This example clears the index for Elasticsearch.

Repair Nodetool in Cassandra Example

This example repairs the Nodetool in Cassandra.

List Available Backups Example

This example lists all backups that are available.

View Details of a Specific Backup Example

This example lists the details of a specific backup.

Release History Table
Starting in CSO 4.1, full backup of all platform, op-co, tenant, and customer data can be run manually or periodically and that data can be restored from the backups when and if the need arises.