Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

 
 

CSO Disaster Recovery

In case of any failures you can recover CSO Release 6.2.0. To recover CSO Release 6.2.0 you must have already taken a backup and saved the backup file.

To recover CSO Release 6.2.0:

  1. Based on the hypervisor you are using, do one of the following:
    • If you are using KVM as the hypervisor:

      1. Copy the CSO 6.2.0 backup folder to the bare metal server.

      2. From the backup folder, copy the _topology.conf file to the Contrail_Service_Orchestration_6.2.0/topology/ folder.

        For example:

      3. Provision the VMs. For information on provisioning KVM hypervisor, see Provision VMs on Contrail Service Orchestration Servers in CSO Installation and Upgrade Guide.

      4. Copy the backup folder file from the bare metal server to the startupserver1 VM.

      5. Log in to the startupserver1 VM as the root user.

      6. Expand the installer package.

        The expanded package is a directory that has the same name as the installer package and contains the installation files.

      7. From the backup folder, copy the _topology.conf file to the Contrail_Service_Orchestration_6.2.0/topology/ folder.

    • If you are using ESXi as the hypervisor:

      1. Copy the backup folder to the startupserver1 VM.

      2. Expand the installer package.

        The expanded package is a directory that has the same name as the installer package and contains the installation files.

      3. From the backup folder, copy the _topology.conf file to the Contrail_Service_Orchestration_6.2.0/topology/ folder in the startupserver1 VM.

        For example:

  2. Run the deploy.sh command.
  3. Run the following command:
  4. Run the pre_disaster recovery script.

    python /usr/local/bin/pre_disaster_recovery.py

    Enter yes at the prompt.

  5. Restore the data from the new backup created in step 3 by using the cso_backupnrestore script.

    where backuppath is the new backup path.

    If the restore procedure fails for any of the above components, you must retry to restore only those components. At times, restore of mariadb fails at the first attempt but is successful at the second attempt.

  6. Synchronize the data between nodes.

    Enter y at the prompt.

  7. Copy the certificate from the backup folder to SDN-based load balancing (SBLB) HA Proxy.
  8. Restart the SBLB HA Proxy.
  9. Copy the certificate from the backup folder to Central HA Proxy.
  10. Restart the Central HA Proxy.
  11. Run the following commands on installer VM to update the Nginx certificates.
  12. Deploy microservices.
  13. Reindex the elastic search.
    1. Open the csp.csp-ems-regional deployment file.

    2. Change the replicas to 2 and increase the memory from 500Mi to 2048Mi (2Gi).

    3. Save the file.

    4. Start the reindex process.

    5. Using the admin token, run the following API to build the policy indices:

  14. Create the RabbitMQ FMPM queue.
  15. Load the data.
  16. Synchronize the Virtual Route Reflector (VRR). Use the admin token. Do not use the cspadmin token.
    1. Obtain the topo-uuid for the VRR.

    2. Synchronize the VRR using the POST https://<ip>/routing-manager/synchronize-vrr API.

  17. Restore the SD-WAN and security reports.

    where backuppath is the new backup path.

  18. Restart all fmpm-provider-api and fmpm-provider-core pods by deleting the existing pods.
  19. Delete all the pods displayed in the previous step.
  20. Restore the Contrail Analytics Node (CAN) database.
    Note:

    You can restore the database only if a backup is available. CAN backup is disabled by default. To include CAN data in the backup, comment out contrail_analytics in the following configuration:

    To restore the CAN configuration database, run the following script:

    To restore the CAN analytics database, perform the following steps:

    The analyticsdb backup files are located at /backups/daily/2021-06-07T06:46:37/central/can/contrail_analytics<x>, where x indicates the contrail analytics node number. The value of x ranges from 1 through 3.

    On all the three contrail analytics nodes:

    • Copy the CAN backup files from the startupserver to each CAN VM:

    • Run the following command on the CAN VMs:

      #The path might be different based on uuid.

After a successful upgrade, CSO Release 6.2.0 is functional and you can log in to the Administrator Portal and the Customer Portal.