Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

 

CSO Disaster Recovery

 

In case of any failures you can recover CSO Release 6.0.0. To recover CSO Release 6.0.0 you must have already taken a backup and saved the backup file.

To recover CSO Release 6.0.0:

  1. Based on the hypervisor you are using, do one of the following:
    • If you are using KVM as the hypervisor:

      1. Copy the CSO 6.0.0 backup folder to the bare metal server.
      2. From the backup folder, copy of the _topology.conf file to the Contrail_Service_Orchestration_6.0.0/topology/folder.

        For example:

        cp /root/backups/backupfordr/2020-06-19T17:27:05/config_backups/_topology.conf /root/Contrail_Service_Orchestration_6.0.0/topology/
      3. Provision the VMs. For information on provisioning KVM hypervisor, see Provision VMs on Contrail Service Orchestration Servers in CSO Installation and Upgrade Guide.
      4. Copy the backup folder file from the bare metal server to the startupserver1 VM.
        user@server>scp -r /root/backups/backupfordr/ startupserver1:
      5. Log in to the startupserver1 VM as the root user.
      6. Expand the installer package.
        root@startupserver1:~/# tar –xvzf Contrail_Service_Orchestration_6.0.0.tar.gz
      7. From the backup folder, copy the _topology.conf file to the Contrail_Service_Orchestration_6.0.0/topology/ folder.
        cp /root/backups/backupfordr/2020-06-19T17:27:05/config_backups/_topology.conf /root/Contrail_Service_Orchestration_6.0.0/topology/

      The expanded package is a directory that has the same name as the installer package and contains the installation files.

    • If you are using ESXi as the hypervisor:

      1. Copy the backup folder to the startupserver1 VM.
      2. Expand the installer package.
        root@startupserver1:~/# tar –xvzf Contrail_Service_Orchestration_6.0.0.tar.gz

        The expanded package is a directory that has the same name as the installer package and contains the installation files.

      3. From the backup folder, copy of the _topology.conf file to the Contrail_Service_Orchestration_6.0.0/topology/ folder in the startupserver1 VM.

        For example:

        cp /root/backups/backupfordr/2020-06-19T17:27:05/config_backups/_topology.conf /root/Contrail_Service_Orchestration_6.0.0/topology/
  2. Run the deploy.sh command.
    root@startupserver1:~/Contrail_Service_Orchestration_6.0.0./deploy.sh
  3. Run the following command to create a new backup:
    cso_backupnrestore -b backup -s backup60new
  4. Run the pre_disaster recovery script.

    python /usr/local/bin/pre_disaster_recovery.py

    Enter yes at the prompt.

  5. Restore the data from the new backup created in step 3 by using the cso_backupnrestore script.

    where backuppath is the new backup path.

    If the restore procedure fails for any of the above components, you must retry to restore only those components. At times, restore of mariadb fails at the first attempt but is successful at the second attempt.

  6. Synchronize the data between the nodes.
    cso_backupnrestore -b nodetool_repair

    Enter y at the prompt.

  7. Copy the certificate from the backup folder to SDN-based load balancing (SBLB) HA Proxy.
    salt-cp -G "roles:haproxy_confd_sblb" /root/backups/backupfordr/2020-06-19T17:27:05/config_backups/haproxycerts/minions/minions/csp-central-proxy_sblb1.NH5XCS.central/files/etc/pki/tls/certs/ssl_cert.pem /etc/pki/tls/certs
    salt-cp -G "roles:haproxy_confd_sblb" /root/backups/backupfordr/2020-06-19T17:27:05/config_backups/haproxycerts/minions/minions/csp-central-proxy_sblb1.NH5XCS.central/files/etc/pki/tls/certs/ssl_cert.crt /etc/pki/tls/certs
  8. Restart the SBLB HA Proxy.
    salt -C "G@roles:haproxy_confd_sblb" cmd.run "service haproxy restart"
  9. Copy the certificate from the backup folder to Central HA Proxy.
    salt-cp -G "roles:haproxy_confd" /root/backups/backupfordr/2020-06-19T17:27:05/config_backups/haproxycerts/minions/minions/csp-central-proxy1.NH5XCS.central/files/etc/pki/tls/certs/ssl_cert.pem /etc/pki/tls/certs
    salt-cp -G "roles:haproxy_confd" /root/backups/backupfordr/2020-10-29T06:45:11/config_backups/haproxycerts/minions/minions/csp-central-proxy1.NH5XCS.central/files/etc/pki/tls/certs/ssl_cert.crt /etc/pki/tls/certs
  10. Restart the Central HA Proxy.
    salt -C "G@roles:haproxy_confd" cmd.run "service haproxy restart"
  11. Run the following commands on the installer VM to update the Nginx certificates.
    kubectl get secret -n central | grep cso-ingress-tls
    kubectl delete secret cso-ingress-tls -n central kubectl create secret tls cso-ingress-tls --key /root/backups/backupfordr/2020-10-29T06:45:11/config_backups/haproxycerts/minions/minions/csp-central-proxy1.NH5XCS.central/files/etc/pki/tls/certs/ssl_cert.key --cert /root/backups/backupfordr/2020-10-29T06:45:11/config_backups/haproxycerts/minions/minions/csp-central-proxy1.NH5XCS.central/files/etc/pki/tls/certs/ssl_cert.crt -n central
  12. Deploy microservices.
    /python.sh micro_services/deploy_micro_services.py
  13. Reindex the elastic search.
    1. Open the csp.csp-ems-regional deployment file.

      kubectl edit deployment -n regional csp.csp-ems-regional
    2. Change the replicas to 2 and increase the memory from 500Mi to 2048Mi (2Gi).

    3. Save the file.

    4. Start the reindex process.

      cso_backuprestore -b reindex
    5. Using the admin token, run the following API to build the policy indices:

      curl --location --request POST 'https://AdminPortalIP/policy-mgmt/_index' \ --header 'x-auth-token: XXXXXXX‘\ --data-raw ‘'
  14. Create the RabbitMQ FMPM queue.
    ./python.sh upgrade/migration_scripts/common/rabbitmq_fmpm_queue_creation.py
  15. Load the data.
    ./python.sh micro_services/load_services_data.py
  16. Synchronize the Virtual Route Reflector (VRR). Use the admin token. Do not use the cspadmin token.
    1. Obtain the topo-uuid for the VRR.

      GET: https://<IP Address>/topology-service/device
    2. Synchronize the VRR using the POST https://<ip>/routing-manager/synchronize-vrr API.

  17. Restore the SD-WAN and security reports.
    cso_backupnrestore -b restore -s backuppath -t '*' -c 'swift_report' -r 'yes'

    where backuppath is the new backup path.

  18. Restart all fmpm-provider-api and fmpm-provider-core pods by deleting the existing pods.
    root@startupserver1:~# kubectl get pods -n central|grep fmpm-provider
  19. Delete all the pods displayed in the previous step.
    kubectl delete pods csp.csp-fmpm-provider-6644bc8b94-7pvfn csp.csp-fmpm-provider-6644bc8b94-c2psl csp.csp-fmpm-provider-6644bc8b94-gzkht csp.csp-fmpm-provider-6644bc8b94-hz8f5csp.csp-fmpm-provider-6644bc8b94-nsqfs csp.csp-fmpm-provider-6644bc8b94-rq9xq csp.csp-fmpm-provider-core-797f7c48c9-7nm8q csp.csp-fmpm-provider-core-797f7c48c9-7zj67 csp.csp-fmpm-provider-core-797f7c48c9-8njsq csp.csp-fmpm-provider-core-797f7c48c9-rh2jr csp.csp-fmpm-provider-core-797f7c48c9-sswbg csp.csp-fmpm-provider-core-797f7c48c9-zvhps
  20. Restore the Contrail Analytics Node (CAN) database. Note

    You can restore the database only if a backup is available. CAN backup is disabled by default. To include CAN data in the backup, comment out contrail_analytics in the following configuration:

    To restore the CAN configuration database, run the following script:

    ./python.sh upgrade/migration_scripts/common/can_migration.py

    To restore the CAN analytics database, perform the following steps:

    The analyticsdb backup files are located at /backups/daily/2021-06-07T06:46:37/central/can/contrail_analytics<x>, where x indicates the contrail analytics node number. The value of x ranges from 1 through 3.

    On all the three contrail analytics nodes:

    • Copy the CAN backup files from the startupserver to each CAN VM:

      rsync -a<can-backup-files>root@<can-ip>:<created-backup-folder>
    • Run the following command on the CAN VMs:

      docker cp 0000/ analytics_database_cassandra_1:/root
    docker exec -it analytics_database_cassandra_1 bash
    mv /root/mc-* /var/lib/cassandra/data/ContrailAnalyticsCql/statstablev4-d5b63590a7f011eba080c3eb6817d254

    #The path might be different based on uuid.

    cd /var/lib/cassandra/data/ContrailAnalyticsCql/statstablev4-d5b63590a7f011eba080c3eb6817d254
    chown -R cassandra:cassandra *
    nodetool -p 7200 refresh -- ContrailAnalyticsCql statstablev4

After a successful upgrade, install the patch. See Installation Instructions to Patch the CSO Release 6.0.0.