Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

Backup and Restore

 

This topic describes the backup and restore capabilities available in Paragon Automation. Although Paragon Automation is a GUI-based application, the backup and restore operations are managed from Paragon Insights cMGD CLI. Postgres is the primary persistent storage database for microservices. Backup files are saved to a local persistent volume on the cluster nodes. The backup procedure can be performed while microservices are running and does not affect the operation of the cluster. However, for restore procedures, microservices are stopped and the cluster is not functional until the databases are restored.

Currently, you cannot schedule a backup or restore procedure. Also, you cannot select applications to be backed up and restored. We have a preconfigured fixed set of applications for each component that can be backed up and restored.

The backup and restore procedures are implemented by containerized scripts that are invoked through Kubernetes jobs.

Figure 1: Backup and Restore Process
Backup and Restore Process

Back Up the Configuration

Data across most Paragon Automation applications is primarily stored in Postgres. When you back up a configuration, system determined predefined data is backed up. When you perform a backup, the operational system and microservices are not affected. You can continue to use Paragon Automation and a backup can run concurrently at the same time.

To back up the current Paragon Automation configuration:

  1. Determine and log in to the cMGD CLI managed by Paragon Insights (formerly Healthbot).

    For example:

    root@primary-node:~# kubectl get -n healthbot pods -l app=mgd
    root@primary-node:~# kubectl exec -it -n healthbot mgd-57b5754b7f-26mlm -- bash
    root@primary-node:~# cli
    Note

    The main interface in the Kubernetes cluster is kubectl which is installed on a primary node. However, you can also access the Kubernetes API from any node that has access to the cluster, including the control host. To use a node other than the primary node, you must ensure that you copy the admin.conf file and set the kubeconfig environment variable, or you can use the export KUBECONFIG=config-dir/admin.conf command.

  2. Enter the request system backup path path-to-backup-folder command to start a backup job which backs up all databases up until the moment you execute the command.

    For example:

    root@mgd-57b5754b7f-26mlm> request system backup path /hello/world/

    A corresponding Kubernetes db-backup-hello-world job is created. The Kubernetes job creates a backup of the predefined data and the files are stored in a local persistent volume.

  3. After backup is complete, you must explicitly and manually back up the EMS resources using kubectl.

    1. Back up jobmanager-identitysrvcreds and devicemodel-connector-default-scope-id:
      root@primary-node:~# kubectl get secrets -n ems jobmanager-identitysrvcreds devicemodel-connector-default-scope-id -o yaml > ems-scope-bkup.yaml
    2. (Optional) If SMTP is configured on the Paragon Automation cluster, then back up the available iam-smtp-config secret:
      root@primary-node:~# kubectl get secrets -n common iam-smtp-config -o yaml > iam-smtp-bkup.yaml

      If this command fails, then SMTP is not configured in the cluster and the error can be ignored.

Frequently Used kubectl Commands to View Backup Details

To view the status of your backup or the location of your backup files, or to view more information on the backup files, use the following commands.

  • Backup jobs exist in the common namespace and use the common=db-backup label. To view all backup jobs:
    root@primary-node:~# kubectl get -n common jobs -l common=db-backup
  • To view more details of a specific Kubernetes job:
    root@primary-node:~# kubectl describe -n common jobs/db-backup-hello-world
  • To view the logs of a specific Kubernetes job:
    root@primary-node:~# kubectl logs -n common --tail 50 jobs/db-backup-hello-world
  • To determine the location of the backup files:
    root@primary-node:~# kubectl get -n common pvc db-backup-pvc

    This will point you to the local persistent volume. Look up that persistent volume to determine the node on which the backup files are stored.

    root@primary-node:~# kubectl describe -n common pv local-pv-cb20f386

    To view all the backup files, log in to the node and navigate to the location of the backup folder.

    root@primary-node:~# ssh root@10.49.xxx.x2

To view commonly seen backup and restore failure scenarios, see Common Backup and Restore Issues.

Restore the Configuration

You can restore a Paragon Automation configuration from a previously backed up configuration folder. A restore operation rewrites the databases with all the backed up configuration information. You cannot selectively restore databases. When you perform a restore, a Kubernetes job is spawned, which will stop the affected microservices. The job then restores the backed-up configuration, and restarts the microservices. Paragon Automation is not functional until the time the restoration procedure is complete.

You cannot run multiple restore jobs at the same time, since microservices are stopped during the restoration process. Also, you cannot run both backup and restore processes concurrently.

Note

We strongly recommend that you restore a configuration during a maintenance window, else the system can go into an insconsistent state.

To restore the Paragon Automation configuration to a previously backed up configuration:

  1. Log in to the cMGD CLI managed by Paragon Insights (formerly Healthbot).

    For example:

    root@primary-node:~# kubectl get -n healthbot pods -l app=mgd
    root@primary-node:~# kubectl exec -it -n healthbot mgd-57b5754b7f-26mlm -- bash
    root@primary-node:~# cli
  2. Enter the request system restore path path-to-backup-folder command to restore the configuration with the files in the specified backup folder on the persistent volume.

    For example:

    root@mgd-57b5754b7f-26mlm> request system restore path /hello/world/

    A corresponding Kubernetes db-restore-hello-world job is created. The restore process takes longer than a backup process since microservices are stopped and restarted. When the restoration is complete, the Paragon Automation system is not operational immediately. You must wait around ten minutes for the system to stabilize and become fully functional.

    Note

    If you are logged in during the restore process, you must log out and log back in after the restore process is complete.

  3. After restore is complete, you must explicitly restore the EMS resources with the previously manually backed-up EMS backup files.

    1. Delete the jobmanager-identitysrvcreds and devicemodel-connector-default-scope-id EMS secrets resources.
      root@primary-node:~# kubectl delete secrets -n ems jobmanager-identitysrvcreds devicemodel-connector-default-scope-id
    2. Restore the previously backed-up EMS resources.
      root@primary-node:~# kubectl apply -f ems-scope-bkup.yaml
    3. Restart the jobmanager and devicemodel-connector EMS services.
      root@primary-node:~# kubectl rollout restart deploy jobmanager devicemodel-connector -n ems
    4. (Optional) If SMTP is configured on the Paragon Automation cluster, delete the current SMTP secrets file and restore from the previously backed up file.
      root@primary-node:~# kubectl delete secret -n common iam-smtp-config
      root@primary-node:~# kubectl apply -f iam-smtp-bkup.yaml
    5. (Optional) Delete the manually backed-up files.
      root@primary-node:~# rm ems-scope-bkup.yaml iam-smtp-bkup.yaml

Frequently Used kubectl Commands to View Restore Details

To view more information and the status of your restore process, use the following commands.

  • Restore jobs exist in the common namespace and use the common=db-restore label. To view all restore jobs:
    root@primary-node:~# kubectl get -n common jobs -l common=db-restore
  • To view more details of a specific Kubernetes job:
    root@primary-node:~# kubectl describe -n common jobs/db-restore-hello-world
  • To view the logs of a particular Kubernetes job:
    root@primary-node:~# kubectl logs -n common --tail 50 jobs/db-restore-hello-world

To view commonly seen backup and restore failure scenarios, see Common Backup and Restore Issues.