Back Up and Restore Paragon Automation

This topic describes the backup and restore functionality available for Paragon Automation.

CAUTION:

We support only the snapshot feature available on VMware clients to back up and restore your cluster. We recommend that you use only the snapshot feature available on VMware clients to back up and restore your cluster.

Back Up and Restore Using VMware Snapshots

You can use the snapshot functionality available on VMware vSphere and ESXi clients to back up and restore your Paragon Automation cluster and application configuration data. The backup procedure can be performed while the microservices and applications are running and does not affect the operation of the network. The snapshot creates a copy of your cluster and application configuration information. The snapshot restore functionality enables you to restore your cluster on the same set of VMs from which the backup was taken. The VMs are suspended before performing a restore operation and the downtime associated with a restore operation is minimal.

Back Up Using VMware Snapshots
Restore Using VMware Snapshots

Back Up Using VMware Snapshots

To back up your Paragon Automation cluster and application configuration data, use any one of the following options:

On VMware vSphere client
1. Log in to any of the cluster nodes.
2. Use the request paragon health-check Paragon Shell command to ensure that your cluster is in good health. The status of the cluster must be GREEN.
3. Log in to the VMware vSphere client which manages the Paragon Automation VMs.
4. Click on your cluster host in the VM inventory.
  
  Your cluster node VMs are listed in the Virtual Machines tab under VMs.
5. Select all the VMs, right-click, and click Take snapshot. The snapshot creation dialog box appears.
6. Enter a name for the snapshot and a description. Also, select the Include virtual machine's memory check-box.
7. Click Create to create the snapshot. The snapshot creation takes a few minutes.
  
  You can check the progress of the job under Recent Tasks or in the Task Console.
On VMware ESXi client
1. Log in to any of the cluster nodes.
2. Use the request paragon health-check Paragon Shell command to ensure that your cluster is in good health. The status of the cluster must be GREEN.
3. Log in to the VMware ESXi server on which your Paragon Automation VMs are located.
4. Select a cluster node VM listed under Virtual Machines, right-click and click Take snapshot. The snapshot creation dialog box appears.
5. Enter a name for the snapshot and a description. Also, select the Include virtual machine's memory check-box.
6. Click Create to create the snapshot.
7. Repeat steps 4 through 6 for the other three VMs and enter appropriate snapshot names when prompted.
  
  Note:
  We recommend that you create snapshots of all the VMs one after the other as close as possible to each other in time.
8. (Optional) You can check the progress of the snapshot jobs under Recent Tasks or in the Task Console.
Using APIs
You can also use the snapshot.create option to take snapshots of VMs using APIs. For more information, see https://github.com/vmware/govmomi/blob/main/govc/USAGE.md#snapshotcreate.

Restore Using VMware Snapshots

To restore your Paragon Automation cluster and application configuration data which was previously backed up in snapshots, use any one of the following options:

On VMware vSphere client
1. Log in to the VMware vSphere client which manages the Paragon Automation VMs and where the snapshots were previously taken.
2. Click your cluster host in the VM inventory.
  
  Your cluster VMs are listed in the Virtual Machines tab under VMs.
3. Select all the VMs, right-click, and click Suspend to suspend the operation of the VMs. Do not restore the VMs when they are powered on.
4. Select a cluster node VM, right-click, and click Manage snapshots. The Manage snapshots page appears with a list of available snapshots.
5. Select the snapshot that you want to restore and click Restore snapshot.
  
  Ensure that you select a snapshot that is recent and stable.
6. Repeat steps 4 through 5 for the other three VMs.
  
  Alternatively, you can select a VM, right-click and click Revert to Latest Snapshot if your immediate last snapshot was the most stable version.
  
  Note:
  We recommend that you restore snapshots of all the VMs one after the other as close as possible to each other in time. Do not wait for the snapshot restoration of any one VM to complete before restoring the other.
7. (Optional) You can check the progress of the job under Recent Tasks or in the Task Console.
8. Once the restore process is complete, wait for a few minutes for the VMs to power-on and the cluster to stabilize.
9. Log in to any of the cluster nodes.
10. Type the show paragon cluster nodes command in Paragon Shell. Ensure that the status of all the nodes is Ready.
11. Type the show paragon cluster pods command and ensure that the status of all the pods is either Running or Completed.
12. Type the request paragon health-check command to ensure that your cluster is in good health. The status of the cluster must be GREEN.
On VMware ESXi client
1. Log in to the VMware ESXi server on which your Paragon Automation VMs are located.
2. Select you cluster node VM listed under Virtual Machines, right-click and click Suspend to suspend the operation of the VM. Do not restore a VM when it is powered on.
3. Right-click the selected VM and click Manage snapshots. The Manage snapshots page appears with a list of available snapshots.
4. Select the snapshot that you want to restore and click Restore snapshot.
  
  Ensure that you select a snapshot that is recent and stable.
5. Repeat steps 2 through 4 for the other three VMs.
  
  Alternatively, you can select a VM, right-click and click Revert to Latest Snapshot if your immediate last snapshot was the most stable version.
  
  Note:
  We recommend that you restore snapshots of all the VMs one after the other as close as possible to each other in time. Do not wait for the snapshot restoration of any one VM to complete before restoring the other.
6. (Optional) You can check the progress of the restore jobs under Recent Tasks or in the Task Console.
7. Once the restore process is complete, wait for a few minutes for the VMs to power-on and the cluster to stabilize.
8. Log in to any of the cluster nodes.
9. Type the show paragon cluster nodes command in Paragon Shell. Ensure that the status of all the nodes is Ready.
10. Type the show paragon cluster pods command and ensure that the status of all the pods is either Running or Completed.
11. Type the request paragon health-check command to ensure that your cluster is in good health. The status of the cluster must be GREEN.
Using APIs
You can also use the snapshot.revert option to restore backed-up snapshots of VMs using APIs. For more information, see https://github.com/vmware/govmomi/blob/main/govc/USAGE.md#snapshotrevert.

Disaster Recovery

You can restore data on a fresh installation from a backup take on a different setup using the procedure described in this section. You can back up your Paragon Automation cluster and all associated application configuration by exporting it to your ESXi server. Once exported, you can redeploy it on a new installation. During backup and re-installation, your cluster is shutdown and is nonoperational. Perform the following steps to export and redeploy:

Verify that the cluster is in good health. Log in to one of the primary nodes and execute the request paragon health-check command from the Linux root shell.

The Overall Cluster Status must be GREEN.
Export your current cluster configuration.
1. Log in to one of the node VMs. You are placed in Paragon Shell.
2. Shut down the cluster using the request paragon shutdown type cluster command.
3. Log in to your ESXi server.
4. Export the configuration of all VMs one after another and enter appropriate names when prompted.
  
  Perform the steps described in the VMware vSphere documentation at Export an OVF Template.
The VM configuration is backed up locally as four sets of OVF, .vmdk, .nvram, and .mf configuration files.
Redeploy the exported configuration as a fresh installation.

Perform the steps described in Create the node VMs. Select the corresponding sets of OVF and configuration files for each VM.

The cluster is re-installed afresh.
(Optional) Verify that the cluster is in good health. Log in to one of the cluster nodes and execute the request paragon health-check command.

The Overall Cluster Status must be GREEN.

Back Up and Restore Using Paragon Shell

You can also use the backup and restore functionality available in Paragon Shell to back up and restore your Paragon Automation cluster and application configuration data.

CAUTION:

The backup and restore functionality available in Paragon Shell is inconsistent and might not work as intended. We recommend that you use the snapshot feature available on VMware clients to back up and restore your cluster. See Back Up and Restore Using VMware Snapshots.

Back Up Using Paragon Shell
Restore Using Paragon Shell
View or Delete Backup Files
Upload or Download Backup Files

Back Up Using Paragon Shell

You can back up your current Paragon Automation network configuration using Paragon Shell CLI. When you run the backup command, all the application configuration information stored in PostgreSQL,ArangoDB, and Airflow configuration database systems are backed up. The backup command also backs up telemetry information stored in TimescaleDB, OpenSearch, and VictoriaMetrics database systems. The backup procedure can be performed while the microservices and applications are running and does not affect the operation of the network.

To back up your Paragon Automation configuration state.

Log in as root user to any of the Paragon Automation nodes.
Execute the following command to back up the configuration.
root@Primary1> request paragon backup
Alternatively, you can use the request paragon backup config command to back up only configuration information or request paragon backup telemetry command to back up only telemetry data.
(Optional) View progress of the backup job. The backup process takes a few minutes to complete. In order to not block the terminal for use for the whole duration of the backup job, you are returned to the command prompt before the backup job is complete. You can view the progress status of the backup job by using the following command.
root@Primary1> show paragon backup status backup-id backup ID

Upon completion, a backup file with a filename in the yyyymmdd-hhmmss format is created. The backup file is stored in the local persistent /export/paragon-shell/backup folder on the node. You'll have to exit out of Paragon Shell to the Linux root shell to navigate to the backup folder.

Caveat of the backup process

Application configurations (such as devices, sites, service orders, and so on) are backed up, but certificates and infrastructure services configurations are not backed up. This information must be kept unchanged before you perform a restore.
Backup process will capture current infrastructure configurations, but information is used for reference only. The same configuration can be used to instantiate a new setup.

Restore Using Paragon Shell

You can restore your Paragon Automation network configuration from a backup configuration file. To restore from a backup configuration file, all microservices and applications must be stopped, and the cluster is not functional until the databases are restored. Once the databases are flushed and restored to the backed-up configuration, the applications must be restarted, and configuration restored from the databases must be reparsed.

To restore your Paragon Automation configuration from a specific backup configuration file.

Execute the following command to uninstall all the running applications.

root@Primary1> request paragon destroy cluster input “-v -t
                                apps”

This command takes some time to complete, and you must wait until all the applications are shut down before proceeding to the next step. To check the status of the applications, perform the following steps.

Type exit to exit Paragon Shell to the Linux root shell.

Check each application namespace for running deployments using the kubectl get deployments -n namespace command, where namespace is the name of the application namespace. Use this command for each namespace and the output of every command should be empty, which indicates that no deployments are running. Table 1 lists all the namespaces that you must check.

Table 1: Namespace List
`airflow`	`epic`	`foghorn`	`gnmi-term`
`mems`	`netbox`	`northstar`	`oc-term`
`paa`	`papi`	`streams`	`trust`
`healthbot` The command output should display only the following deployments: `vm-operator-victoria-metrics-operator, vmauth-victoria-metrics-auth, vminsert-victoria-metrics-cluster`

When all applications are shut down, you are ready to proceed with the restore process.

Type cli to log in again to Paragon Shell.

(Optional) If your backed up file is in a remote location, download the file locally.
root@Primary1> request paragon backup download backup-id backup-ID storage-location path user username password password
Where:

path is the path to the remote backup file.

username and password are the login credentials to the remote server.
Restore the applications configuration from the backup file.
root@Primary1> request paragon restore backup-id backup-ID
(Optional) View progress status of the restore job.
root@Primary1> show paragon restore status backup-id backup-ID
Reinstall all the applications.
root@Primary1> request paragon deploy cluster input “-v -t apps”
Reparse and synchronize the restored configuration.
root@Primary1> request paragon restore sync

You must perform the restore operation only on the node on which the required backup file is located.

Caveats of the restore process

When you perform the restore operation, the network configuration is returned to the configuration present in the backup file. From the time the backup was taken, if the network configuration has changed due to new devices being onboarded or new service orders being executed, the network configuration in Paragon Automation might be different from the actual network state. To ensure that the network configuration in Paragon Automation and the actual network state have minimal mismatch post a restore operation, we recommend that you take regular periodic backups or specific backups after every network intent change.
You cannot restore data from a release different from the current installed release of Paragon Automation.
You can restore data on the same setup on which a backup was taken and not on a fresh installation using the procedure described in this section. To restore backed up data on a fresh installation, use the procedure described in Disaster Recovery.
Since a backup does not store the certificates and infrastructure services configurations, that information must be kept unchanged during restoration.
Resources allocated to the network won’t be preserved after a restore and you must ensure that you release the allocated resources during the window between taking a backup and performing a restore.

View or Delete Backup Files

To view a list of all backup files across all nodes, use the following command:

root@Primary1> show paragon
                        backup

The node connects to all the other nodes in the Paragon Automation cluster using SSH and displays a list of all backup file names along with the IP address of the node on which the file is located.

To view a list of backup files along with a list of failed backup attempts, use the following command:

root@Primary1> show paragon backup
                        include-failure
                        true

To delete a backup file, use the following command.

root@Primary1> request paragon backup delete
                        backup-id backup-ID

You can delete a backup file that is located only on the node on which you execute the command.

Upload or Download Backup Files

Upload your backed-up file to a remote location outside the Paragon Automation cluster, within the same network as the cluster or in a different network. To upload your backup file to a remote location, use the following command:

root@Primary1>request paragon backup upload backup-id backup-ID storage-location scp://IP:port/remote-path user username password password

To view progress of the backup file upload command, use the following command:

root@Primary1> show paragon backup upload status backup-id backup-ID

To download your backup file from a remote location, use the following command:

root@Primary1>request paragon backup download backup-id backup-ID storage-location scp://IP:port/remote-path user username password password

To view progress of the backup file download command, use the following command: