Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

 

CSO Disaster Recovery

 

In case of any failures you can recover CSO Release 5.1.2. To recover CSO Release 5.1.2 you must have already taken a backup and saved the backup file.

To recover CSO Release 5.1.2:

  1. Based on the hypervisor you are using, do one of the following:
    • If you are using KVM as the hypervisor:

      1. Copy the CSO 5.1.2 backup folder to the bare metal server.
      2. From the backup folder, copy of the topology.conf file to the Contrail_Service_Orchestration_5.1.2/topology/folder.

        For example:

        cp /root/backups/backupfordr/2020-06-19T17:27:05/config_backups/_topology.conf /root/Contrail_Service_Orchestration_5.1.2/topology/
      3. Provision the VMs. For information on provisioning KVM hypervisor, see Provision VMs on Contrail Service Orchestration Servers in CSO Installation and Upgrade Guide.
      4. Copy the backup folder file from the bare metal server to the startupserver_1 VM.
        user@server>scp csoVersion.tar.gz root@<startupserver_>IP>:/root/
      5. Log in to the startupserver_1 VM as the root user.
      6. Run the get_vm_details.shscript to find the IP address of the startupserver_1 VM.
      7. Access the shell of the startupserver_1 VM.
      8. Expand the installer package.
        root@host:~/# tar –xvzf csoVersion.tar.gz

      The expanded package is a directory that has the same name as the installer package and contains the installation files.

      Go to Step 2.

    • If you are using ESXi as the hypervisor:

      1. Copy the backup folder to the startupserver_1 VM.
      2. From the backup folder, copy of the topology.conf file to the Contrail_Service_Orchestration_5.1.2/topology/folder in the startupserver_1 VM.

        For example:

        cp /root/backups/backupfordr/2020-06-19T17:27:05/config_backups/_topology.conf /root/Contrail_Service_Orchestration_5.1.2/topology/
      3. Log in to the startupserver_1 VM as the root user.
      4. Run the get_vm_details.sh script to find the IP address of the startupserver_1 VM
      5. Access the shell of the startupserver_1 VM.
      6. Expand the installer package.
        root@host:~/# tar –xvzf csoVersion.tar.gz

      The expanded package is a directory that has the same name as the installer package and contains the installation files.

  2. Run the deploy.sh command.
    root@host:~/Contrail_Service_Orchestration_5.1.2./ deploy.sh

    Use the interactive script to create configuration files for the environment specific topology. Because you have taken a backup, the answers are prepopulated for each question in the interactive script.

    Note

    Ensure that you specify no for the following question:

    Whether this is an upgrade setup?

  3. Restore the data by using the cso_backupnrestore script.

    If the restore procedure fails for any of the above components, you must retry to restore only those components.

  4. Restore Cassandra.

    To restore Cassandra:

    1. Run python /usr/local/bin/cassandra_token_update.py to update token and cluster details.
      : root@server:# python /usr/local/bin/cassandra_token_update.py
    2. Restore Cassandra.
      cso_backupnrestore -b restore -s backuppath -t '*' -c 'cassandra' -r 'yes'
  5. Synchronize the data between nodes.
    cso_backupnrestore -b nodetool_repair
  6. Copy the certificate from the backup folder to SDN-based load balancing (SBLB) HA Proxy.
    salt-cp -G "roles:haproxy_confd_sblb" /root/backups/config_backups/haproxycerts/minions/minions/csp-regional-sblb1.TNL2OQ. regional/files/etc/pki/tls/certs/ssl_cert.pem /etc/pki/tls/certs salt-cp -G "roles:haproxy_confd_sblb" /root/backups/config_backups/haproxycerts/minions/minions/csp-regional-sblb1.TNL2OQ. regional/files/etc/pki/tls/certs/ssl_cert.crt /etc/pki/tls/certs
  7. Restart the SBLB HA Proxy.
    salt -C "G@roles:haproxy_confd_sblb" cmd.run "service haproxy restart"
  8. Copy the certificate from the backup folder to Central HA Proxy.
    salt-cp -G "roles:haproxy_confd" /root/backups/config_backups/haproxycerts/minions/minions/csp-central-lbvm1.HBLGHQ. central/files/etc/pki/tls/certs/ssl_cert.pem /etc/pki/tls/certs salt-cp -G "roles:haproxy_confd" /root/backups/config_backups/haproxycerts/minions/minions/csp-central-lbvm1.HBLGHQ. central/files/etc/pki/tls/certs/ssl_cert.crt /etc/pki/tls/certs
  9. Restart the Central HA Proxy.
    salt -C "G@roles:haproxy_confd" cmd.run "service haproxy restart"
  10. Run the following commands on installer VM to update the Nginx certificates.
    kubectl get secret -n central | grep cso-ingress-tls content_copyzoom_out_map
    kubectl delete secret cso-ingress-tls -n central kubectl create secret tls cso-ingress-tls --key /root/backups/config_backups/haproxycerts/minions/minions/csp-central-lbvm1.5R8JKN. central/files/etc/pki/tls/certs/ssl_cert.key --cert /root/backups/config_backups/haproxycerts/minions/minions/csp-central-lbvm1.5R8JKN. central/files/etc/pki/tls/certs/ssl_cert.crt -n central
  11. Run the upgrade.sh script.
  12. Restore the SD-WAN and security reports.
    cso_backupnrestore -b restore -s backuppath -t '*' -c 'swift_report' -r 'yes'
  13. Restart all fmpm-provider-core pods by deleting the existing pods.
    root@startupserver1:~# kubectl get pods -n central|grep fmpm-provider-core
    root@startupserver1:~# kubectl delete pod csp.csp-fmpm-provider-core-647ff6598d-4qxxd csp.csp-fmpm-provider-core-647ff6598d-94wdx csp.csp-fmpm-provider-core-647ff6598d-dt6vj csp.csp-fmpm-provider-core-647ff6598d-hbnw2 csp.csp-fmpm-provider-core-647ff6598d-mx8fn csp.csp-fmpm-provider-core-647ff6598d-zd2zt -n central
  14. Restore Contrail Analytics Node (CAN) database.
    ./python.sh upgrade/migration_scripts/common/can_migration.py

    The analyticsdb backup files are located at /root/backups/backup411/2020-05-21T00:43:50/central/can/can<x>

    ssh root@<new-can-ip-[123]>
    docker cp 0000/mc-* analyticsdb:/root
    docker exec -it analyticsdb bash
    mv /root/mc-* /var/lib/cassandra/data/ContrailAnalyticsCql/statstablebystrtagv3-c5e9b4c056f711ea8a948909f467ce30

    #The path may be different based on uuid #The path may be different based on uuids

    cd /var/lib/cassandra/data/ContrailAnalyticsCql/statstablebystrtagv3-c5e9b4c056f711ea8a948909f467ce30
    chown -R cassandra:cassandra *
    nodetool refresh -- ContrailAnalyticsCql statstablebystrtagv3

After a successful upgrade, CSO Release 5.1.2 is functional and you can log in to the Administrator Portal and the Customer Portal.