Known Issues

This section lists known issues in Juniper Networks CSO Release 4.0.2.

AWS Spoke

• The AWS device activation process takes up to 30 minutes. If the process does not complete in 30 minutes, a timeout might occur and you must retry the process. You do not need to download the cloud formation template again.

  Procedure

  To retry the process:

  1. Log in to Customer Portal.
  2. Access the Activate Device page, enter the activation code, and click Next.
  3. After the CREATE_COMPLETE message is displayed on the AWS server, click Next on the Activate Device page to proceed with device activation.

  Bug Tracking Number: CXU-19102.

CSO HA

• In an HA setup, users are not able to log into CSO for about five minutes after one of the central servers hosting the HAproxy VRRP master has been brought down.

  Workaround: There is no known workaround.

  Bug Tracking Number: CXU-28255

• In an HA setup, users are not able to log into CSO when the Contrail server that hosts the MariaDB master has been brought down. This is because the MariaDB cluster is unhealthy when the server that hosts the MariaDB master is down. CSO recovers when the MariaDB master is back online.

  Workaround: There is no known workaround.

  Bug Tracking Number: CXU-27178

• After a power failure, CAN installed on a physical server does not come up online correctly.

  Workaround:

  Procedure

  Follow these steps to restore CAN installed on a physical server:

  1. Log in to the CAN server and scp the /root/can_bkp folder to installer VM.
  2. Reimage the server.
  3. Navigate to the Contrail_Service_Orchestration_4.0.2 folder.
  4. Execute DEPLOYMENT_ENV=central ./deploy_infra_services.sh on installer VM until it starts deploying NTP.
  5. Execute salt '*contrail*' network.hw_addr eth0.

     csp-contrailanalytics-3.4D5UTX.central: 52:54:00:2c:a6:4d

     csp-contrailanalytics-1.4D5UTX.central: 52:54:00:2b:4f:da

     csp-contrailanalytics-2.4D5UTX.central: 52:54:00:ea:ee:67

  6. Open deployments/central/roles.conf. Search for can1, can2 and can3. Make sure that the MAC addresses listed in (4) matches the field hardware_address for each of can1, can2 and can3.
  7. Open deployment/central/topology.conf, and edit servers under [TARGETS] servers = csp-contrailanalytics-1, csp-contrailanalytics-2, csp-contrailanalytics-3.
  8. Execute DEPLOYMENT_ENV=central ./deploy_infra_services.sh.
  9. Check health of CAN by executing ./components_health.sh.
  10. To restore the data back, copy the backed up can_bkp folder from installer VM to the respective CAN servers under root/.
  11. Execute the following steps on all three CAN servers:

      Procedure

      1. root@sspt-ubuntu5-vm7:~# docker exec controller service cassandra stop
      2. root@sspt-ubuntu5-vm7:~# docker exec analyticsdb service cassandra stop
      3. root@sspt-ubuntu5-vm7:~# docker exec -it controller bash
      4. root@sspt-ubuntu5-vm7(controller):/var/lib/cassandra# rm -rf *
      5. root@sspt-ubuntu5-vm7(controller): exit
      6. root@sspt-ubuntu5-vm7:~# docker exec -it analyticsdb bash
      7. root@sspt-ubuntu5-vm7(analyticsdb):/var/lib/cassandra# rm -rf *
      8. root@sspt-ubuntu5-vm7(analyticsdb): exit
      9. root@sspt-ubuntu5-vm7:~# cd can_bkp/analyticsdb_old/
      10. root@sspt-ubuntu5-vm7:~/can_bkp/analyticsdb_old/cassandra# docker cp commitlog/ analyticsdb:/var/lib/cassandra
      11. root@sspt-ubuntu5-vm7:~/can_bkp/analyticsdb_old/cassandra# docker cp data/ analyticsdb:/var/lib/cassandra
      12. root@sspt-ubuntu5-vm7:~/can_bkp/analyticsdb_old/cassandra# docker cp saved_caches/ analyticsdb:/var/lib/cassandra
      13. root@sspt-ubuntu5-vm7:~# cd can_bkp/controller_old/
      14. root@sspt-ubuntu5-vm7:~/can_bkp/controller_old/cassandra# docker cp commitlog/ controller:/var/lib/cassandra
      15. root@sspt-ubuntu5-vm7:~/can_bkp/controller_old/cassandra# docker cp data/ controller:/var/lib/cassandra
      16. root@sspt-ubuntu5-vm7:~/can_bkp/controller_old/cassandra# docker cp saved_caches/ controller:/var/lib/cassandra
      17. root@sspt-ubuntu5-vm7:~# docker exec controller chown -R cassandra:cassandra /var/lib/cassandra/
      18. root@sspt-ubuntu5-vm7:~# docker exec analyticsdb chown -R cassandra:cassandra /var/lib/cassandra/
      19. root@sspt-ubuntu5-vm7:~# docker exec analyticsdb service cassandra start
          
      20. root@sspt-ubuntu5-vm7:~# docker exec controller service cassandra start
  12. Check health of CAN by executing ./components_health.sh.

• As part of VRR recovery process in case of power failure, a tenant named recovery is created for restoring the VRR configuration. However, if the configuration that needs to be recovered is huge, the recovery tenant creation times out and fails even though the configuration is successfully restored to VRR in due course.

  Workaround: There is no workaround required as the configuration is usually restored to VRR even if the recovery tenant creation has timed out.

  Bug Tracking Number: CXU-27197

• In a CSO HA environment, two RabbitMQ nodes are clustered together, but the third RabbitMQ node does not join the cluster. This might occur just after the initial installation, if a virtual machine reboots, or if a virtual machine is powered off and then powered on.

  Workaround: Do the following:

  Procedure

  1. Log in to the installer VM.
  2. Navigate to the current deployment directory for CSO—for example, /root/Contrail_Service_Orchestration_4.0.2/.
  3. Execute the ./recovery.sh command.
  4. Specify the option to recover RabbitMQ and press Enter.
  5. In the RabbitMQ dashboards for the central and regional microservices VMs, confirm that all the available infrastructure nodes are present in the cluster.

  Bug Tracking Number: CXU-12107

• When a high availability (HA) setup comes back up after a power outage, MariaDB instances do not come back up on the VMs.

  Workaround:

  Procedure

  Perform the following steps to recover the MariaDB instances:

  1. Log in to the installer VM.
  2. Navigate to the current deployment directory for CSO; for example, /root/Contrail_Service_Orchestration_4.0.2/.
  3. Execute the sed -i "s@/var/lib/mysql/grastate.dat@/mnt/data/mysql/grastate.dat@g" recovery/components/recover_mariadb.py command
  4. Execute the ./recovery.sh command.
  5. Specify the option to recover MariaDB and press Enter.

  Bug Tracking Number: CXU-20260

• In some cases, when power fails, the ArangoDB cluster does not form.

  Workaround:

  Procedure

  1. Log in to the centralinfravm3 VM.
  2. Execute the following commands:
     service arangodb3.cluster stop

     cd /var/lib/arangodb3 && mv setup.json setup.json.old
  3. Log in to the centralinfravm2 VM.
  4. Execute the following commands:
     service arangodb3.cluster stop

     cd /var/lib/arangodb3 && mv setup.json setup.json.old
  5. Log in to the centralinfravm1 VM.
  6. Execute the following commands:
     service arangodb3.cluster stop

     cd /var/lib/arangodb3 && mv setup.json setup.json.old
  7. On the centralinfravm1 VM, execute the service arangodb3.cluster start command and wait for 20 seconds for the command to finish executing.
  8. On the centralinfravm2 VM, execute the service arangodb3.cluster start command and wait for 20 seconds for the command to finish executing.
  9. On the centralinfravm3 VM, execute the service arangodb3.cluster start command and wait for 20 seconds for the command to finish executing.

  Bug Tracking Number: CXU-20346

• In a HA setup, if you shut down all the CSO servers, after the servers are restarted successfully, MariaDB and ArangoDB fail to form their respective clusters.

  Workaround:

  Procedure

  1. Perform a clean reboot of the central infrastructure VMs.
  2. After the VMs have rebooted successfully, check the cluster health on the HAproxy page (http://central-ip-address:1936, where central-IP-address is the IP address of the VM that hosts the microservices for the central POP).
  3. If the MariaDB and ArangoDB clusters are still down, you can recover the clusters by performing the following procedures:

     • To recover the MariaDB cluster, perform the following steps:

       Procedure

       1. On the centralinfravm1 VM, execute the service mysql stop command.
       2. On the centralinfravm2 VM, execute the service mysql stop command.
       3. On the centralinfravm3 VM, execute the service mysql stop command.
       4. On all three central infrastructure VMs, verify that the service has stopped executing the service mysql status command.
       5. On the centralinfravm1 VM, start the service by executing the service mysql start command.
       6. On the centralinfravm2 VM, start the service by executing the service mysql start command.
       7. On the centralinfravm3 VM, start the service by executing the service mysql start command.
       8. On all three central infrastructure VMs, verify that the service has started executing the service mysql status command.

     • To recover the ArangoDB cluster, perform the following steps:

       Procedure

       1. On the centralinfravm1 VM, execute the service arangodb3.cluster stop command.
       2. On the centralinfravm2 VM, execute the service arangodb3.cluster stop command.
       3. On the centralinfravm3 VM, execute the service arangodb3.cluster stop command.
       4. On all three central infrastructure VMs, verify that the service has stopped executing the ps -aef|grep arangodb command.
       5. On the centralinfravm1 VM, start the service by executing the service arangodb3.cluster start command.
       6. On the centralinfravm2 VM, start the service by executing the service arangodb3.cluster start command.
       7. On the centralinfravm3 VM, start the service by executing the service arangodb3.cluster start command.
       8. On all three central infrastructure VMs, verify that the service has started executing the ps -aef|grep arangodb command.

  Bug Tracking Number: CXU-21819.

• In a HA setup, if you onboard devices and deploy policies on the devices and if one of the policy deployments is in progress when a microservices or infrastructure node goes down, the deployment job is stuck in the In Progress state for about 90 minutes (the default timeout value), and you cannot perform deploy operations for the tenant for about 90 minutes.

  Workaround: Wait for the job to fail and then redeploy the policy.

  Bug Tracking Number: CXU-21922

• If an infrastructure node goes down in a HA setup in which all nodes were previously up, and you create a firewall policy and try to deploy the policy, the deployment job is stuck in the in-progress state and a Redis timeout error is displayed in the job log.

  Workaround:

  Procedure

  1. Wait for approximately 90 minutes for the job to fail.
  2. Bring up the infrastructure node that was down.
  3. Redeploy the firewall policy.

  Bug Tracking Number: CXU-24559

• While you are upgrading CSO (Production Environment with HA) from Release 3.3.1 to Release 4.0.2, the upgrade fails after a snapshot is taken because the regional Kubernetes node is in the Not Ready status.

  Workaround: Restart the Docker service.

  Procedure

  1. On the regional K8 master node, run the following commands:

     • service docker stop

     • rm -rf /var/lib/docker/*

     • service docker start

  2. Run the ./upgrade.sh script.

  Bug Tracking Number: CXU-25625

• You cannot access the Administration Portal login page if the flannel network subnet is changed.

  Workaround:

  1. Log in to central microservices VMs.

  2. In all central microservices VMs, run the following commands in parallel:

     service flanneld stop

     service docker stop

     service flanneld start

     sleep 10

     service docker start

     After executing these commands, wait for 10 minutes.

  3. On any one of the central microservices VM, run the following command to delete all pods:

     kubectl delete pods --all --force --grace-period=0 -n central

  Bug Tracking Number: CXU-23736

SD-WAN

• Link affinity does not work when there are multiple links with the same cost. There may be frequent switch between equal cost links and that might cause network flapping.

  Workaround: Ensure that the cost values are different for each of the links.

  Bug Tracking Number: CXU-27969

• On the Site SLA Performance page, applications with different SLA scores are plotted at the same coordinate on the x-axis.

  Workaround: None.

  Bug Tracking Number: CXU-19768

• When all local breakout links are down, site to Internet traffic fails even though there is an active overlay to the hub.

  Workaround: None.

  Bug Tracking Number: CXU-19807

• If the Internet breakout WAN link of the cloud hub is not used for provisioning the overlay tunnel by at least one spoke site in a tenant, then traffic from sites to the Internet is dropped.

  Workaround: Ensure that you configure a firewall policy to allow traffic from security zone trust-tenant-name to zone untrust-wan-link, where tenant-name is the name of the tenant and wan-link is the name of the Internet breakout WAN link.

• Bug Tracking Number: CXU-21291

• On the SD-WAN Events page, for link switch events, if you mouse over the Reason field, the values displayed for the SLA metrics are the ones that are recorded when the system logs are sent from the device and not the values for which the SLA violation was detected.

  Workaround: None.

  Bug Tracking Number: CXU-21461

• In a hub-and-spoke topology with multitenancy enabled, when a spoke site is configured with two MPLS and two Internet links with MPLS selected as the default, traffic from the hub to the spoke site takes the same path instead of taking the path (link) on which the traffic was received by the hub (incoming WAN link). However, there is no traffic loss.

  Workaround: Remove the static route with the next hop and replace it with a static route with the qualified next hop.

  Bug Tracking Number: CXU-23197

• If a WAN link on a CPE device goes down, the WAN tab of the Site-Name page (in Administration Portal) displays the corresponding link metrics as N/A.

  Workaround: None.

  Bug Tracking Number: CXU-23996

• If a tenant has a real-time-optimized site, link switch events (on the Monitor page) might display the same WAN link for both source and destination tunnels.

  Workaround: None.

  Bug Tracking Number: CXU-24154

• If you delete a cloud hub that is created in Release 3.3.1, CSO does not delete the stage-2 configuration.

  Workaround: You must manually delete the stage-2 configuration from the device.

  Bug Tracking Number: CXU-25764

Security Management

• When a certificate renewal is triggered from the VPN Authentication page under the Certificate tab, the certificate table becomes empty after renewing the certificate.

  Workaround: Refresh the Certificate page to display the required certificate details.

  Bug Tracking Number: CXU-25561

• On the Active Database page in Customer Portal, the wrong installed device count is displayed. The count displayed is for all tenants and not for a specific tenant.

  Workaround: None.

  Bug Tracking Number: CXU-20531

• If a cloud hub is used by two tenants, one with public key infrastructure (PKI) authentication enabled and other with preshared key (PSK) authentication enabled, the commit configuration operation fails. This is because only one IKE gateway can point to one policy and if you define a policy with a certificate then the preshared key does not work.

  Workaround: Ensure that the tenants sharing a cloud hub use the same type of authentication (either PKI or PSK) as the cloud hub device.

  Bug Tracking Number: CXU-23107

• If UTM Web-filtering categories are installed manually (by using the request system security UTM web-filtering category install command from the CLI) on an NFX150 device, the intent-based firewall policy deployment from CSO fails.

  Workaround: Uninstall the UTM Web-filtering category that you installed manually by executing the request security utm web-filtering category uninstall command on the NFX150 device and then deploy the firewall policy.

  Bug Tracking Number: CXU-23927

• On the Identity Management page, if you click Download JIMS, the Juniper Identity Management Service (JIMS) software is downloaded in HTML format.

  Workaround: Download the JIMS software from the Download Software page.

  Bug Tracking Number: CXU-24278

• If SSL proxy is configured on a dual CPE device and if the traffic path is changed from one node to another node, the following issue occurs:

  • For cacheable applications, if there is no cache entry the first session might fail to establish.

  • For non-cacheable applications, the traffic flow is impacted.

  Workaround: None.

  Bug Tracking Number: CXU-25526

• The UTM policy configuration is not deployed on an SD-WAN site with the SRX device model SRX345-DUAL-AC.

  Workaround:

  1. Add the SRX345-DUAL-AC device model to the schema file.

     Note In the schema-svc docker, the schema file is available at /opt/csp-schema-data/*configuration.json.

  2. Restart the pod.

  Bug Tracking Number: CXU-25706

Site and Tenant Workflow

• Porting of cloud hub sites to tenants fails if the cloud hub site names exceed 15 characters.

  Workaround: Ensure that cloud hub site names do not exceed 15 characters even though you can have cloud hub site names of up to 256 characters in the global instance.

  Bug Tracking Number: CXU-28078

• ZTP for NFX150 may fail before creating the vlink. Though ZTP goes through successfully on retry, service chain activation may fail.

  Workaround: Delete and add the site so that ZTP goes through successfully in one attempt.

  Bug Tracking Number: CXU-27967

• During site activation, activation of NFX250 dual CPE connected to MX series cloud hub device may fail with the following error message: No existing device_initiated device connection.

  Workaround: Retry the failed ZTP job from the administration portal.

  Bug Tracking Number: CXU-27902

• After a site upgrade, status of policies that are associated with the site appears as pending deployment even though they are already deployed.

  Workaround: Trigger a policy deployment job to deploy the policies. CSO does not deploy the policies unless there are updates to the policy, but the status of policies are appropriately updated after you run a deployment job.

  Bug Tracking Number: CXU-27528

• SLA profiles created by a tenant are not deleted when the tenant is deleted.

  Workaround: There is no known workaround.

  Bug Tracking Number: CXU-27054

• If you create a new tenant with the name of a tenant that was deleted, certain inconsistencies such as policy deployment failure are noticed.

  Workaround: When you create a tenant, ensure that you do not use the same name as that of a deleted tenant.

  Bug Tracking Number: CXU-26886

• Site upgrade for hub sites that were created using custom device profile or cloned device profile is incomplete.

  Workaround:

  • After the upgrade, go to tssm core docker by entering the following command: docker exec -it docker name bash

  • In the docker run the following command: root@csp:/# cd /opt/meta_data/

  • From /opt/meta_data, run cp SRX_Advanced_SDWAN_HUB_option_1_upgrade.yaml custom_device_profile_upgrade.yaml

  Bug Tracking Number: CXU-26532

• The tenant delete operation fails when CSO is installed with an external Keystone.

  Workaround: You must manually delete the tenant from the Contrail OpenStack user interface.

  Bug Tracking Number: CXU-9070

• If you try to activate a branch SRX Series device with the factory-default configuration, the stage-1 configuration commit might fail when there are active DHCP server bindings on the device. This is because of the default DHCP server settings present in factory-default configuration.

  Workaround: When you are pre-staging the CPE device for activation, remove the DHCP server-related configuration from the device by executing the following commands on the Junos OS CLI:

  set system services dhcp-local-server group jdhcp-group interface fxp0.0

  set system services dhcp-local-server group jdhcp-group interface irb.0

  Bug Tracking Number: CXU-13446

• In some cases, if automatic license installation is enabled in the device profile, after ZTP is complete, the license might not be installed on the CPE device even though license key is configured successfully.

  Workaround: Reinstall the license on the CPE device by using the Licenses page on the Administration Portal.

  Bug Tracking Number: PR1350302.

• For a tenant, LAN segments with overlapping IP prefixes across sites are not supported.

  Workaround: Create LAN segments with unique IP prefixes across sites for the tenant.

  Bug Tracking Number: CXU-20494

• When the primary and backup interfaces of the CPE device uses the same WAN interface of the hub, the backup underlay might be used for Internet or site-to-site traffic even though the primary links are available.

  Workaround: Ensure that you connect the WAN links of each CPE device to unique WAN links of the hub.

  Bug Tracking Number: CXU-20564

• After you configure a site, you cannot modify the configuration either before or after activation.

  Workaround: None.

  Bug Tracking Number: CXU-21165

• On the Configure Site page, the values that you specify for the time zone and the IP address of the NTP server are not being pushed to the device.

  Workaround: Configure the NTP server IP address and the time zone on the device, manually:

  1. Log in to the device.

  2. In the configuration mode, run the following commands:

     • set system ntp server IP address

     • set system time-zone time zone

  3. Commit the changes

  Bug Tracking Number: CXU-23971

• On an NFX250 device, if you disable (detach) a failed service successfully and then try to delete the site, the site is not deleted.

  Workaround: None.

  Bug Tracking Number: CXU-24355

• When you try to activate a site with an SRX Series device, ZTP might fail with an error during the installation of the default trusted certificates.

  Workaround: Retry the failed job after some time.

  Bug Tracking Number: CXU-24487

• If you try to activate a site with an MPLS link by using DHCP, the default route pointing to the MPLS gateway is added to the hub device, which results in Internet traffic from the hub taking the MPLS link.

  Workaround: None.

  Bug Tracking Number: CXU-24666

• If you trigger the tenant creation workflow, the tenant might be displayed in the CSO GUI even before the job is completed. If you then try to trigger workflows for that tenant, the subsequent jobs fail because the tenant creation job is not completed.

  Workaround: Wait for the tenant creation job to complete successfully before triggering any workflows for the tenant.

  Bug Tracking Number: CXU-24783

• The Configure Site operation for a cloud spoke site fails.

  Workaround: None.

  Bug Tracking Number: CXU-24795

• The Configure Site operation fails if you import a cloud hub with a name that is different from that of other tenants.

  Workaround: While you are importing a cloud hub, specify the same name that is used while onboarding a cloud hub for a global service provider.

  Bug Tracking Number: CXU-25740

• You cannot configure a site with dual CPE devices if WAN links are used exclusively for local breakout traffic.

  Workaround: While you are creating a site and enabling the link for local breakout, instead of selecting the Use only for breakout traffic option, select Use for breakout & WAN traffic. Also, while you are configuring a site ensure that the WAN link is connected to a hub.

  Bug Tracking Number: CXU-25776

General

• If a link or connectivity to CSO flaps during a site upgrade or image upgrade, the upgrade job remains stuck in the in progress state.

  Workaround: Use Rest APIs to mark the job as failed and the site status as provisioned, and then retry the upgrade job.

  Bug Tracking Number: CXU-27726

• In an HA setup, the Synchronize_Device_Inventory job that was triggered as part of load_service remains incomplete in an unknown state. However, this does not impact any of the workflows.

  Workaround: There is no known workaround.

  Bug Tracking Number: CXU-28004

• When users rebuild a small deployment with the UI installer and using custom-generated certificates, the underlay tunnels for the NFX250 device remain down if there is a mismatch between the default host name and the custom-generated host name.

  Workaround: There is no known workaround.

  Bug Tracking Number: CXU-27976

• If you upload images with the same filename for two different device families, the file gets overwritten.

  Workaround: Rename the files before uploading to CSO.

  Bug Tracking Number: CXU-27713

• If multiple sites are using the same MX series cloud hub, IPSec overlay tunnels for some of the WAN links may fail to come up and show the following error: Negotiation failed with error code NO_PROPOSAL_CHOSEN received from peer (5 times).

  Workaround: Clear the IPSec session from the connected MX series cloud hub by executing the clear services ipsec-vpn ipsec security-associations command.

  Bug Tracking Number: CXU-27638

• For spokes connected to MX hub, OAM tunnels are displayed in the data overlay section of the Monitor > Overview page and the Sites > WAN pages of the administration portal. This does not have any functional impact.

  Workaround: There is no known workaround.

  Bug Tracking Number: CXU-27449

• After a cloud hub device is rebooted, the device connectivity to CSO is lost. This problem occurs in CSO Release 4.0.2 that has been upgraded from CSO Release 3.3.1 and has the stage-2 configuration applied.

  Workaround: Disable and enable lo0 on the device.

  Procedure

  To disable and enable the lo0 interface, log in to the cloud hub device and run the following commands from the configuration mode:

  1. set interface lo0 disable
  2. commit
  3. delete interface lo0 disable
  4. commit and quit

  Bug Tracking Number: CXU-27420

• ZTP for SRX devices fails. This problem occurs if the SRX device was connected to clients on the LAN side before ZTP and has bindings that are not cleared during ZTP.

  Workaround: There is no known workaround.

  Bug Tracking Number: CXU-27376

• ZTP of NFX-250 over PPPoE fails and incomplete configuration is pushed to the device.

  Workaround: Use links other than PPPoE for ZTP of NFX-250.

  Bug tracking number: CXU-27357

• On NFX models NFX150-C-S1, NFX150-C-S1-AE, and NFX150-C-S1-AA, VNFs that require more than 4 GB of memory (for example, Riverbed, Fortinet) fail to launch.

  Workaround: For Ubuntu VNF, you can reduce the memory requirement from CSO to enable launching the VNF.

  Bug Tracking Number: CXU-27180

• In high-availability setups where the login page and logo have been updated, the old login image gets displayed occasionally when the page is refreshed.

  Workaround: Restart the csp.admin-portal-ui Docker images.

  Bug Tracking Number: CXU-27037

• SRX cluster devices are not listed in the device selection window for pushing licenses to devices.

  Workaround: For SRX cluster devices, manually install the license from the device CLI.

  Bug tracking number: CXU-26654

• Reverting CSO from 4.0.2 to 3.3.1 fails when CAN is unhealthy.

  Workaround:

  Run the following commands:

  root@canvm:~# docker exec -it analyticsdb bash

  root@canvm(analyticsdb):/# service cassandra status

  If Cassandra is not running, go to /var/log/cassandra/debug.log and look for error messages similar to:DEBUG [main] 2018-09-27 01:01:34,857 CommitLogReader.java:223 - Replaying /var/lib/cassandra/commitlog/CommitLog-6-1537967618037.log (CL version 6, messaging version 10, compression null) ERROR [main] 2018-09-27 01:01:35,660 CommitLogReader.java:214 Exiting due to error while processing commit log during initialization.

  Delete the commit logs that are causing the error:

  root@canvm(analyticsdb):/# rm /var/lib/cassandra/commitlog/CommitLog-6-1537967618037.log

  Run the following command to restart Cassandra: root@canvm(analyticsdb):/# service cassandra restart

  Log in to controller container, using the docker exec -it controller bash command, and follow the preceding steps to check whether Cassandra is running in controller container and to delete the logs if necessary.

  On high availability setups, repeat this on all nodes.

  Run ./components_health.sh on the installer VM to ensure CAN is healthy.

  Bug Tracking Number: CXU-26456

• In an SD-WAN, traffic from LAN to WAN stops after a single-legged Ubuntu VNF has been brought up. This problem occurs because of port cross connect between the left interface of VNF and GWR interface (on NFX250) or Flowd tap interface (for NFX150).

  Workaround: Remove port cross connect configuration. You can do it either on the device or by using Stage 2 template.

  Bug Tracking Number: CXU-26282

• GRE tunnel fails to come up online when a new site is added. This problem occurs because the new site uses the PPPoE IP that was originally-assigned to the tunnel even though the tunnel IP has since changed because of renegotiation by PPPoE.

  Workaround: Use the fixed IPs (through CHAP/PAP authentication or CPE Mac address) for clients when you use PPPoE.

  Bug Tracking Number: CXU-26606

• On an Ubuntu VNF spawned on an NFX250 device, the ping command to a website address (fully qualified domain name) does not work.

  Workaround:

  Procedure

  1. In Resource Designer, clone the existing ubuntu-fw-NFX250 template for the NFX250 device.
  2. Edit the template and ensure that offloads are disabled for the Left Interface.
  3. Click Next and complete the edit operation.

  Bug tracking number: CXU-24985

• GWR may fail to come up after an NFX device image has been upgraded.

  Workaround:

  Procedure

  Follow these steps to manually restart the VNF from JDM:

  1. Find the GWR VNF name using the virsh list command.
  2. Start the GWR VNF using the virsh start GWR-VNF-Name command.

  Bug Tracking Number: CXU-24823

• If you create VNF instances in the Contrail cloud by using Heat Version 2.0 APIs, a timeout error occurs after 120 instances are created.

  Workaround: Contact Juniper Networks Technical Support.

  Bug Tracking Number: CXU-15033

• The provisioning of CPE devices fails if all VRRs within a redundancy group are unavailable.

  Workaround: Recover the VRR that is down and retry the provisioning job.

  Bug Tracking Number: CXU-19063

• After the upgrade, the health check on the standalone Contrail Analytics Node (CAN) fails.

  Workaround:

  Procedure

  1. Log in to the CAN VM.
  2. Execute the docker exec analyticsdb service contrail-database-nodemgr restart command.
  3. Execute the docker exec analyticsdb service cassandra restart command.

  Bug Tracking Number: CXU-20470

• The load services data operation or health check of the infrastructure components might fail if the data in the Salt server cache is lost because of an error.

  Workaround: If you encounter a Salt server-related error, do the following:

  Procedure

  1. Log in to the installer VM.
  2. Execute the salt '*' deployutils.get_role_ips 'cassandra' command to confirm whether one or more Salt minions have lost the cache.

     • If the output returns the IP address for all the Salt minions, this means that the Salt server cache is fine; proceed to step 7.

     • If the IP address for some minions is not present in the output, this means that the Salt server has lost its cache for those minions and must be rebuilt as explained from step 3.

  3. Navigate to the current deployment directory for CSO; for example, /root/Contrail_Service_Orchestration_4.0.2/.
  4. Redeploy the central infrastructure services (up to the NTP step):

     Procedure

     1. Execute the DEPLOYMENT_ENV=central ./deploy_infra_services.sh command.
     2. Press Ctrl+c when you see the following message on the console:

        2018-04-10 17:17:03 INFO utils.core Deploying roles set(['ntp']) to servers ['csp-central-msvm', 'csp-contrailanalytics-1', 'csp-central-k8mastervm', 'csp-central-infravm']

  5. Redeploy the regional infrastructure services (up to the NTP step):

     Procedure

     1. Execute the DEPLOYMENT_ENV=regional ./deploy_infra_services.sh command.
     2. Press Ctrl+c when you see a message similar to the one for the central infrastructure services.
  6. Execute the salt '*' deployutils.get_role_ips 'cassandra' command and confirm that the output displays the IP addresses of all the Salt minions.
  7. Re-run the load services data operation or the health component check that had previously failed.

  Bug Tracking Number: CXU-20815

• For an MX Series cloud hub device, if you have configured the Internet link type as OAM_and_DATA, the reverse traffic fails to reach the spoke device if you do not configure additional parameters by using the Junos OS CLI on the MX Series device.

  Workaround:

  Procedure

  1. Log in to the MX Series device and access the Junos OS CLI.
  2. Find the next-hop-service outside-service-interface multiservices interface as follows:

     Procedure

     1. Execute the show configuration | display set | grep outside-service-interface command.
     2. In the output of the command, look for the multiservices (ms-) interface corresponding to the service set that CSO created on the device.

        The name of the service set is in the format ssettenant-name_DefaultVPN-tenant-name, where tenant-name is the name of the tenant.

        The following is an example of the command and output:

        show configuration | display set | grep outside-service-interface

        set groups mx-hub-Acme-Acme_DefaultVPN-vpn-routing-config services service-set ssetAcme_DefaultVPN-Acme next-hop-service outside-service-interface ms-1/0/0.4008

        In this example, the tenant name is Acme and the multiservices interface used is ms-1/0/0.4008.

  3. After you determine the correct interface, add the following configuration on the device: set routing-instances WAN_0 interface ms-interface

     where ms-interface is the name of the multiservices interface obtained in the preceding step.

  4. Commit the configuration.

  Bug Tracking Number: CXU-21818

• In Resource Designer, if you add a VNF that does not require a password and trigger the Add VNF Manager workflow, you are asked to enter a password even though the VNF does not require it.

  Workaround: Even for VNFs that do not require a password, enter a dummy password in Resource Designer when you are creating a VNF package.

  Bug Tracking Number: CXU-21845.

• In a full mesh topology, the simultaneous deletion of LAN segments on all sites is not supported.

  Workaround: Delete LAN segments on one site at a time.

  Bug Tracking Number: CXU-21936

• When you install the CSO Downloader app on MacOS, you might receive an error message indicating that the application cannot be opened because it is from an unidentified developer.

  Workaround: Access the MacOS Security & Privacy settings and allow the CSO Downloader app to be opened and continue with the installation.

  Bug Tracking Number: CXU-22661

• If you run the script to revert an upgraded CSO Release 4.0.0 setup to CSO Release 3.3.1, the revert operation fails because of an ArangoDB cluster error.

  Workaround: Use the same workaround as CXU-20346.

  Bug Tracking Number: CXU-23338

• On a CSO setup with secure OAM configured, if you bring up the FortiGate VNF and then apply the license on the VNF, the VNF reboots. However, after rebooting, sometimes the VNF does not come back up.

  Workaround: To ensure that the VNF comes back up, deactivate the VNF and then reactivate it by performing the following steps:

  Procedure

  1. Log in to the JDM CLI of the NFX Series device and access configuration mode.
  2. Deactivate the VNF by executing the deactivate virtual-network-functions Fortinet-oob-2-Firewall command.
  3. Commit the changes by executing the commit command.
  4. Rollback the changes by executing the rollback 1 command
  5. Commit the changes by executing the commit command.
  6. Exit the configuration mode by executing the quit command.
  7. Execute the show virtual-network-functions command and confirm that the status is Running alive, which means that the VNF is up.

  Bug Tracking Number: CXU-23371.

• If one or more VRRs are down, jobs might take a long time to complete, or, in some cases, fail.

  Workaround: Ensure that all VRRs are up before trying the Add Tenant or Add Site workflows.

  Bug Tracking Number: CXU-23710

• The image upgrade of the vSRX gateway router on NFX Series devices by using the CSO GUI is not supported.

  Workaround: Upgrade the image by using the CLI of the NFX Series device.

  Bug Tracking Number: CXU-23804.

• On an NFX Series device with a Ubuntu VNF instantiated, if you use SSH to do log in to the VNF by using the loopback IP address (configured for secure OAM) with port 49154, the connection does not work.

  Workaround:

  Procedure

  1. Use SSH to log in to the vSRX gateway router by using the loopback IP address.
  2. Use SSH to log in to the OAM IP address of the Ubuntu VNF with username root and password passw0rd.
  3. In the Ubuntu VNF, add a route to the IP address of the machine from where you want to log in by using SSH.

     You can now use SSH to log in from the configured machine by using the loopback IP address with port 49154.

  Bug Tracking Number: CXU-23953

• If you are using the GUI installer to install CSO, sometimes the installation page freezes (percentage completion on the VMs does not change) during the installation because of a Rest API timeout.

  Workaround: Reload the CSO installation page in the browser, which will update the status of the installation.

  Bug Tracking Number: CXU-24471

• When you reboot a device from the Tenant Devices or Devices pages, the reboot job fails because the connectivity is lost during the reboot.

  Workaround: Check the operational status of the device on the Tenant Devices or Devices page. During the reboot phase, the operational status of the device is Down. After the device is successfully rebooted and connectivity is restored, the operational status of the device changes to Up. You can now trigger operations on the device by using the CSO GUI.

  Bug Tracking Number: CXU-24512

• If you are using the GUI installer to install CSO, sometimes the UI freezes during the installation and no installation progress is seen. However, the installation continues in the backend.

  Workaround: Perform the following tasks:

  Procedure

  1. Reload the installation UI page in the browser.

     If the UI page loads successfully, no further action is needed. If the UI page does not load, proceed to step 2.

  2. Log in to the installer VM as root.
  3. Kill the existing processes triggered by the GUI installer by executing the kill $(sudo lsof -t -i:8080) command.
  4. Navigate to the /root/cso_dl/Contrail_Service_Orchestration_4.0.2/ directory.
  5. Restart the Flask server by executing the bash run_ui.sh command.
  6. After you see the ==== INFO Installer App initialized ===== message on the console, reload the installation UI page in the browser.

  Bug Tracking Number: CXU-24552

• For an NFX250 device, the Ubuntu VNF service chain configuration is incorrect if you set SINGLE_SSH_TO_NFX to False and then instantiate a service.

  Workaround: None.

  Bug Tracking Number: CXU-25018

• In CSO Release 4.0.1, while you are converting a hub that is created in Release 3.3.1 to a secure OAM hub by using the stage-2 template, the job fails even though the device configuration is updated.

  Workaround: Roll back the stage-2 configuration, save the stage-2 configuration, and redeploy.

  Bug Tracking Number: CXU-25531

• The upgrade from CSO Release 3.3.1 to Release 4.0.1 fails, because the pods do not get deleted.

  Workaround: Delete the pods and rerun the upgrade.sh script.

  1. Log in to the microservice VM.

  2. Execute the kubectl delete deployments,svc,pods,ds,events --all --grace-period=0 --force command.

  3. After the command is successfully executed, rerun the upgrade.sh script.

  Bug Tracking Number: CXU-25737

• On the Audit Logs page, Username and Role columns do not display the actual name and the role of the user, respectively. Instead, the name of the user is displayed as Admin and role of the user is displayed as _member_.admin.

  Workaround: None.

  Bug Tracking Number: CXU-25189

• An error occurs while EEPROM contents for copper ports are being read.

  Workaround: None.

  Bug Tracking Number: PR1372217

• For a device that is provisioned for an OpCo tenant, the software image upgrade fails if you try to upgrade the software image from the Device Images page.

  Workaround: None

  Bug Tracking Number: CXU-25663

• Because of insufficient buffer size, vSRX performs queue scheduling incorrectly and drop packets.

  Workaround: Set the buffer size to 3000 microseconds by executing the set class-of-service schedulers scheduler-name buffer-size temporal 3000 command.

  Bug Tracking Number: PR1361720.