Known Issues

This section lists known issues in Juniper Networks CSO Release 4.0.0.

AWS Spoke

• The AWS device activation process takes up to 30 minutes. If the process does not complete in 30 minutes, a timeout might occur and you must retry the process. You do not need to download the cloud formation template again.

  Procedure

  To retry the process:

  1. Log in to Customer Portal.
  2. Access the Activate Device page, enter the activation code, and click Next.
  3. After the CREATE_COMPLETE message is displayed on the AWS server, click Next on the Activate Device page to proceed with device activation.

  Bug Tracking Number: CXU-19102.

CSO HA

• In a CSO HA environment, two RabbitMQ nodes are clustered together, but the third RabbitMQ node does not join the cluster. This might occur just after the initial installation, if a virtual machine reboots, or if a virtual machine is powered off and then powered on.

  Workaround: Do the following:

  Procedure

  1. Log in to the installer VM.
  2. Navigate to the current deployment directory for CSO—for example, /root/Contrail_Service_Orchestration_4.0.0/.
  3. Execute the ./recovery.sh command.
  4. Specify the option to recover RabbitMQ, and press Enter.
  5. In the RabbitMQ dashboards for the central and regional microservices VMs, confirm that all the available infrastructure nodes are present in the cluster.

  Bug Tracking Number: CXU-12107

• In an HA setup, the time configured for the CAN VMs might not be synchronized with the time configured for the other VMs in the setup. This can cause issues in the throughput graphs.

  Procedure

  Workaround:

  1. Log in to can-vm1 as the root user.
  2. Modify the /etc/ntp.conf file to point to the desired NTP server.
  3. Restart the NTP process.

  After the NTP process restarts successfully, can-vm2 and can-vm3 automatically resynchronize their times with can-vm1.

  Bug Tracking Number: CXU-15681.

• When a high availability (HA) setup comes back up after a power outage, MariaDB instances do not come back up on the VMs.

  Workaround:

  Procedure

  Perform the following steps to recover the MariaDB instances:

  1. Log in to the installer VM.
  2. Navigate to the current deployment directory for CSO—for example, /root/Contrail_Service_Orchestration_4.0.0/.
  3. Execute the sed -i "s@/var/lib/mysql/grastate.dat@/mnt/data/mysql/grastate.dat@g" recovery/components/recover_mariadb.py command.
  4. Execute the ./recovery.sh command.
  5. Specify the option to recover MariaDB, and press Enter.

  Bug Tracking Number: CXU-20260.

• In some cases, when power fails, the ArangoDB cluster does not form.

  Workaround:

  Procedure

  1. Log in to the centralinfravm3 VM.
  2. Execute the following commands:
     service arangodb3.cluster stop

     cd /var/lib/arangodb3 && mv setup.json setup.json.old
  3. Log in to the centralinfravm2 VM.
  4. Execute the following commands:
     service arangodb3.cluster stop

     cd /var/lib/arangodb3 && mv setup.json setup.json.old
  5. Log in to the centralinfravm1 VM.
  6. Execute the following commands:
     service arangodb3.cluster stop

     cd /var/lib/arangodb3 && mv setup.json setup.json.old
  7. On the centralinfravm1 VM, execute the service arangodb3.cluster start command and wait for 20 seconds for the command to finish executing.
  8. On the centralinfravm2 VM, execute the service arangodb3.cluster start command and wait for 20 seconds for the command to finish executing.
  9. On the centralinfravm3 VM, execute the service arangodb3.cluster start command and wait for 20 seconds for the command to finish executing.

  Bug Tracking Number: CXU-20346.

• In a HA setup, if you shut down all the CSO servers, after the servers are restarted successfully, MariaDB and ArangoDB fail to form their respective clusters.

  Workaround:

  Procedure

  1. Perform a clean reboot of the central infrastructure VMs.
  2. After the VMs have rebooted successfully, check the cluster health on the HAproxy page (http://central-ip-address:1936, where central-IP-address is the IP address of the VM that hosts the microservices for the central POP).
  3. If the MariaDB and ArangoDB clusters are still down, you can recover the clusters by performing the following procedures:

     • To recover the MariaDB cluster, perform the following steps:

       Procedure

       1. On the centralinfravm1 VM, execute the service mysql stop command.
       2. On the centralinfravm2 VM, execute the service mysql stop command.
       3. On the centralinfravm3 VM, execute the service mysql stop command.
       4. On all three central infrastructure VMs, verify that the service has stopped executing the service mysql status command.
       5. On the centralinfravm1 VM, start the service by executing the service mysql start command.
       6. On the centralinfravm2 VM, start the service by executing the service mysql start command.
       7. On the centralinfravm3 VM, start the service by executing the service mysql start command.
       8. On all three central infrastructure VMs, verify that the service has started executing the service mysql status command.

     • To recover the ArangoDB cluster, perform the following steps:

       Procedure

       1. On the centralinfravm1 VM, execute the service arangodb3.cluster stop command.
       2. On the centralinfravm2 VM, execute the service arangodb3.cluster stop command.
       3. On the centralinfravm3 VM, execute the service arangodb3.cluster stop command.
       4. On all three central infrastructure VMs, verify that the service has stopped executing the ps -aef|grep arangodb command.
       5. On the centralinfravm1 VM, start the service by executing the service arangodb3.cluster start command.
       6. On the centralinfravm2 VM, start the service by executing the service arangodb3.cluster start command.
       7. On the centralinfravm3 VM, start the service by executing the service arangodb3.cluster start command.
       8. On all three central infrastructure VMs, verify that the service has started executing the ps -aef|grep arangodb command.

  Bug Tracking Number: CXU-21819.

• In a HA setup, if you onboard devices and deploy policies on the devices and if one of the policy deployments is in progress when a microservices or infrastructure node goes down, the deployment job is stuck in the In Progress state for about 90 minutes (the default timeout value), and you cannot perform deploy operations for the tenant for about 90 minutes.

  Workaround: Wait for the job to fail and then redeploy the policy.

  Bug Tracking Number: CXU-21922.

• If an infrastructure node goes down in a HA setup in which all nodes were previously up, and you create a firewall policy and try to deploy the policy, the deployment job is stuck in the in-progress state and a Redis timeout error is displayed in the job log.

  Workaround:

  Procedure

  1. Wait for approximately 90 minutes for the job to fail.
  2. Bring up the infrastructure node that was down.
  3. Redeploy the firewall policy.

  Bug Tracking Number: CXU-24559.

• When you execute the upgrade.sh script to upgrade a setup running CSO Release 3.3.1 to CSO 4.0.0, the load service data operation fails and a 401, Authentication required error is displayed in the upgrade log.

  Workaround: On the installer VM, execute the upgrade.sh script again. The upgrade completes successfully.

  Bug Tracking Number: CXU-24574.

SD-WAN

• On the Site SLA Performance page, applications with different SLA scores are plotted at the same coordinate on the x-axis.

  Workaround: None.

  Bug Tracking Number: CXU-19768.

• When all local breakout links are down, site to Internet traffic fails even though there is an active overlay to the hub.

  Workaround: None.

  Bug Tracking Number: CXU-19807

• If the Internet breakout WAN link of the cloud hub is not used for provisioning the overlay tunnel by at least one spoke site in a tenant, then traffic from sites to the Internet is dropped.

  Workaround: Ensure that you configure a firewall policy to allow traffic from security zone trust-tenant-name to zone untrust-wan-link, where tenant-name is the name of the tenant and wan-link is the name of the Internet breakout WAN link.

• Bug Tracking Number: CXU-21291.

• On the SD-WAN Events page, for link switch events, if you mouse over the Reason field, the values displayed for the SLA metrics are the ones that are recorded when the system logs are sent from the device and not the values for which the SLA violation was detected.

  Workaround: None.

  Bug Tracking Number: CXU-21461.

• In a hub-and-spoke topology with multi-tenancy enabled, when a spoke site is configured with two MPLS and two Internet links with MPLS selected as the default, the traffic from the hub to the spoke site takes the same path instead of taking the path (link) on which the traffic was received by the hub (incoming WAN link). However, there is no traffic loss.

  Workaround: Remove the static route with the next hop and replace it with a static route with the qualified next hop.

  Bug Tracking Number: CXU-23197.

• If a WAN link on a CPE device goes down, the WAN tab of the Site-Name page (in Administration Portal) displays the corresponding link metrics as N/A.

  Workaround: None.

  Bug Tracking Number: CXU-23996.

• If a tenant has a real-time-optimized site, link switch events (on the Monitor page) might display the same WAN link for both source and destination tunnels.

  Workaround: None.

  Bug Tracking Number: CXU-24154.

Security Management

• On the Active Database page in Customer Portal, the wrong installed device count is displayed. The count displayed is for all tenants and not for a specific tenant.

  Workaround: None.

  Bug Tracking Number: CXU-20531.

• If a cloud hub is used by two tenants, one with public key infrastructure (PKI) authentication enabled and other with preshared key (PSK) authentication enabled, the commit configuration operation fails. This is because only one IKE gateway can point to one policy and, if you define a policy with a certificate, then the preshared key does not work.

  Workaround: Ensure that the tenants sharing a cloud hub use the same type of authentication (either PKI or PSK) as the cloud hub device.

  Bug Tracking Number: CXU-23107.

• If UTM Web-filtering categories are installed manually (by using the request system security UTM web-filtering category install command from the CLI) on an NFX150 device, the intent-based firewall policy deployment from CSO fails.

  Workaround: Uninstall the UTM Web-filtering category that you installed manually by executing the request security utm web-filtering category uninstall command on the NFX150 device and then deploy the firewall policy.

  Bug Tracking Number: CXU-23927.

• In the JIMS-to-CSO Configuration panel of the Identity Management page, if you try set a password for the Juniper Identity Management Service (JIMS) user and then save the password, an error message is displayed and the password is not saved.

  Workaround: None.

  Bug Tracking Number: CXU-24419.

• Even though SD-WAN, firewall, or SSL proxy policies are deployed successfully on the device, the CSO GUI incorrectly indicates that policies need to be deployed.

  Workaround: None.

  Bug Tracking Number: CXU-24628.

Site and Tenant Workflow

• The tenant delete operation fails when CSO is installed with an external Keystone.

  Workaround: You must manually delete the tenant from the Contrail OpenStack user interface.

  Bug Tracking Number: CXU-9070

• If you try to activate a branch SRX Series device with the factory-default configuration, the stage-1 configuration commit might fail when there are active DHCP server bindings on the device. This is because of the default DHCP server settings present in factory-default configuration.

  Workaround: When you are pre-staging the CPE device for activation, remove the DHCP server-related configuration from the device by executing the following commands on the Junos OS CLI:

  set system services dhcp-local-server group jdhcp-group interface fxp0.0

  set system services dhcp-local-server group jdhcp-group interface irb.0

  Bug Tracking Number: CXU-13446

• In some cases, if automatic license installation is enabled in the device profile, after ZTP is complete, the license might not be installed on the CPE device even though license key is configured successfully.

  Workaround: Reinstall the license on the CPE device by using the Licenses page on the Administration Portal.

  Bug Tracking Number: PR1350302.

• For a tenant, LAN segments with overlapping IP prefixes across sites are not supported.

  Workaround: Create LAN segments with unique IP prefixes across sites for the tenant.

  Bug Tracking Number: CXU-20494.

• When the primary and backup interfaces of the CPE device uses the same WAN interface of the hub, the backup underlay might be used for Internet or site-to-site traffic even though the primary links are available.

  Workaround: Ensure that you connect the WAN links of each CPE device to unique WAN links of the hub.

  Bug Tracking Number: CXU-20564.

• After you configure a site, you cannot modify the configuration either before or after activation.

  Workaround: None.

  Bug Tracking Number: CXU-21165

• If you initiate the RMA workflow on an NFX Series device that was successfully onboarded and provisioned with stage-2 templates, the device RMA operation might get stuck in the device activation stage if the stage-2 configuration templates have interdependencies.

  Workaround: Ensure that the stage-2 templates that are deployed on the device do not have interdependencies before initiating the device RMA workflow.

  Bug Tracking Number: CXU-21464.

• On the Monitor > Overview page, if you click a site indicating that a major alarm was triggered (site icon color turns orange), and in the subsequent popup, click the link for major alarms in the Alerts & Alarms section, you are taken to the Alarms page. However, no alarm for the device is displayed.

  Workaround: None.

  Bug Tracking Number: CXU-21828.

• If a tenant is deleted and a different tenant is added with the same name as the previously deleted tenant, ZTP of the NFX Series spoke device fails during the VRR reconfiguration.

  Workaround:

  Procedure

  1. Delete the tenant.
  2. Add the tenant with a name that is different from that of the deleted tenant.
  3. Retry the ZTP workflow.

  Bug Tracking Number: CXU-24260.

• On an NFX250 device, if you disable (detach) a failed service successfully and then try to delete the site, the delete site operation fails.

  Workaround: None.

  Bug Tracking Number: CXU-24355.

• When you try to activate a site with an SRX Series device, ZTP might fail with an error during the installation of the default trusted certificates.

  Workaround: Retry the failed job after some time.

  Bug Tracking Number: CXU-24487.

• If you try to delete a tenant that has custom roles defined, the delete tenant operation fails and an error message is displayed in the job log. In addition, though the job fails, the tenant might not be displayed in scope switcher or on the Tenants page.

  Workaround: Delete the custom roles associated with the tenant and then trigger the tenant delete operation.

  Bug Tracking Number: CXU-24655.

• If you try to activate a site with an MPLS link by using DHCP, the default route pointing to the MPLS gateway is added to the hub device, which results in Internet traffic from the hub taking the MPLS link.

  Workaround: None.

  Bug Tracking Number: CXU-24666.

• On the Import Sites page, the operation to import multiple sites by using a JSON file fails.

  Workaround: Use the Sites page to create sites.

  Bug Tracking Number: CXU-24730.

• For an NFX150 device in a hub-and-spoke topology, if you configure the LTE link for OAM traffic, ZTP might fail during the site activation task.

  Workaround: Retry the failed job from the Jobs page, and the ZTP operation is successful.

  Bug Tracking Number: CXU-24762.

• If you trigger the tenant creation workflow, the tenant might be displayed in the CSO GUI even before the job is completed. If you then try to trigger workflows for that tenant, the subsequent jobs fail because the tenant creation job is not completed.

  Workaround: Wait for the tenant creation job to complete successfully before triggering any workflows for the tenant.

  Bug Tracking Number: CXU-24783.

• If you provision a device, push a license on the device (by using CSO), and then try to delete the site (associated with the device), the operation to delete the site fails.

  Workaround: Delete the license from the device (by using CSO) and then delete the site.

  Bug Tracking Number: CXU-24790.

• The Configure Site operation for a cloud spoke site fails.

  Workaround: None.

  Bug Tracking Number: CXU-24795.

• If you try to delete an operating company that has custom roles defined, the delete operation fails. In addition, the operating company might not be displayed in scope switcher or on the Operating Companies (OpCos) page.

  Workaround: Delete the custom roles associated with the operating company and then trigger the deletion of the operating company.

  Bug Tracking Number: CXU-25062.

General

• If you create VNF instances in the Contrail cloud by using Heat Version 2.0 APIs, a timeout error occurs after 120 instances are created.

  Workaround: Contact Juniper Networks Technical Support.

  Bug Tracking Number: CXU-15033

• When you upgrade the gateway router by using the CSO GUI, after the upgrade completes and the gateway router reboots, the gateway router configuration reverts to the base configuration and loses the IPsec configuration added during Zero Touch Provisioning (ZTP).

  Workaround: Before you upgrade the gateway router by using the CSO GUI, ensure that you do the following:

  Procedure

  1. Log in to the Juniper Device Manager (JDM) CLI of the NFX Series device.
  2. Execute the virsh list command to obtain the name of the gateway router (GWR_NAME).
  3. Execute the request virtual-network-functions GWR_NAME restart command, where GWR_NAME is the name of the gateway router obtained in the preceding step.
  4. Wait a few minutes for the gateway router to come back up.
  5. Log out of the JDM CLI.
  6. Proceed with the upgrade of the gateway router by using the CSO GUI.

  Bug Tracking Number: CXU-11823.

• CSO might not come up after a power failure.

  Workaround:

  Procedure

  1. Log in to the installer VM.
  2. Navigate to the /root/Contrail_Service_Orchestration_4.0.0/ directory.
  3. Run the reinitialize_pods.py script as follows:
     ./python.sh recovery/components/reinitialize_pods.py
  4. SSH to the VRR by using the VRR IP address to check if you are able to access the VRR.

     If there is an error in connecting (port 22: Connection refused), then you must recover the VRR by following step 5 through 21.

  5. Log in to physical server hosting the VRR.
  6. Execute the virsh destroy vrr command to destroy the VRR.

     Warning Do not execute the virsh undefine vrr command because doing so will cause the VRR configuration to be lost and the configuration cannot be recovered.

  7. Delete the VRR image that is located in the /root/ubuntu_vm/vrr/vrr-15.1R6.7.qcow2 directory.
  8. Copy the fresh VRR image from the /root/disks/vrr-15.1R6.7.qcow2 directory to the /root/ubuntu_vm/vrr/vrr-15.1R6.7.qcow2 directory.
  9. Execute the virsh start vrr command and wait for approximately 5 minutes for the command to finish executing.
  10. Execute the virsh list –all command to check if the VRR is running or not.

      If the VRR is not running, check that the image that was copied was the uncorrupted image and re-try the steps from step 7.

  11. If the VRR is running, navigate to the /root/ubuntu_vm/vrr/ directory.
  12. Run the ./vrr.exp command to push the base configuration to the VRR.
  13. Check if the VRR is reachable from the regional microservices VM. If the VRR is reachable, proceed to step 14. If the VRR is not reachable:

      Procedure

      1. Log in to the VRR.
      2. Check if the base configuration was pushed properly:

         • If the base configuration was pushed properly, re-check if the VRR is reachable from the regional microservices VM. If the VRR is reachable, proceed to step 14.

         • If the base configuration was not pushed properly:

           Procedure

           1. Add the necessary routes to reach CSO.
           2. Re-check if the VRR is reachable from the regional microservices VM. If the VRR is reachable, proceed to 14.
  14. Import the POP by using the URL https://central-ms-ip:443/tssm/import-pop, where central-ms-ip is the IP address of the central microservices VM.
  15. Use POSTMAN to import the VRR.

      Note Do not import the VRR until the VRR is reachable from the regional microservices VM.

      The following is the JSON format for the VRR. (In the JSON below, <vrr-ip-address> is the IP address of the VRR and <vrr-password> is the password that was configured for the VRR.

      {                "input": {                                "job_name_prefix": "ImportPop",                                "pop": [{                                                "dc_name": "regional",                                                "device": [{                                                                "name": "vrr-<vrr-ip-address>",                                                                "family": "VRR",                                                                "device_ip": "<vrr-ip-address>",                                                                "assigned_device_profile": "VRR_Advanced_SDWAN_option_1",                                                                "authentication": {                                                                                "password_based": {                                                                                                "username": "root",                                                                                                "password": "<vrr-password>"                                                                                }                                                                },                                                                "management_state": "managed",                                                                "pnf_package": "null"                                                }],                                                "name": "regional"                                }]                }}

  16. Verify whether the VRR is imported properly:

      Procedure

      1. Log in to the CSO Administration Portal.
      2. Click Resources > POPs > Import POPs > Import History and confirm that the ImportPop job is running and that it has completed successfully.
  17. On the Tenants page, add a tenant named recovery.
  18. After the tenant is successfully created, log in to the VRR and access the Junos OS CLI.
  19. Execute the show configuration|display set and verify that the tenant configuration (for the previously-configured tenants) is recovered.
  20. Execute the show bgp summary and check that the BGP status to the hub and spokes are Established.
  21. If the status is Not Established, add the routes for the OAM traffic of the hub and spokes to the VRR and recheck the status.

  Bug Tracking Number: CXU-16530

• The provisioning of CPE devices fails if all VRRs within a redundancy group are unavailable.

  Workaround: Recover the VRR that is down and retry the provisioning job.

  Bug Tracking Number: CXU-19063

• The CSO health check displays the following error message: ERROR: ONE OR MORE KUBE-SYSTEM PODS ARE NOT RUNNING

  Workaround:

  Procedure

  1. Log in to the central microservices VM.
  2. Execute the kubectl get pods –namespace=kube-system command.
  3. If the kube-proxy process is not in the Running state, execute the kubectl apply –f /etc/kubernetes/manifests/kube-proxy.yaml command.

     Bug Tracking Number: CXU-20275.

• After the upgrade, the health check on the standalone Contrail Analytics Node (CAN) fails.

  Workaround:

  Procedure

  1. Log in to the CAN VM.
  2. Execute the docker exec analyticsdb service contrail-database-nodemgr restart command.
  3. Execute the docker exec analyticsdb service cassandra restart command.

  Bug Tracking Number: CXU-20470.

• The class-of-service scheduler configuration does not take effect on the CPE device.

  Workaround:

  Procedure

  1. Log in to the CPE device and access the Junos OS CLI.
  2. Enable the scheduler map on each physical interface manually by executing the following commands:
     set class-of-service interfaces interface-name unit * scheduler-map scheduler-map-name

     set interfaces interface-name per-unit-scheduler

     Where interface-name is the name of the physical interface (for example, ge-0/0/4), and scheduler-map-name is the name of the scheduler map.

  3. Commit the configuration on the CPE device.

  Bug Tracking Number: CXU-20708.

• The load services data operation or health check of the infrastructure components might fail if the data in the Salt server cache is lost because of an error.

  Workaround: If you encounter a Salt server-related error, do the following:

  Procedure

  1. Log in to the installer VM.
  2. Execute the salt '*' deployutils.get_role_ips 'cassandra' command to confirm whether one or more Salt minions have lost the cache.

     • If the output returns the IP address for all the Salt minions, this means that the Salt server cache is fine; proceed to step 7.

     • If the IP address for some minions is not present in the output, this means that the Salt server has lost its cache for those minions and must be rebuilt as explained from step 3.

  3. Navigate to the current deployment directory for CSO; for example, /root/Contrail_Service_Orchestration_4.0.0/.
  4. Redeploy the central infrastructure services (up to the NTP step):

     Procedure

     1. Execute the DEPLOYMENT_ENV=central ./deploy_infra_services.sh command.
     2. Press Ctrl+c when you see the following message on the console:

        2018-04-10 17:17:03 INFO utils.core Deploying roles set(['ntp']) to servers ['csp-central-msvm', 'csp-contrailanalytics-1', 'csp-central-k8mastervm', 'csp-central-infravm']

  5. Redeploy the regional infrastructure services (up to the NTP step):

     Procedure

     1. Execute the DEPLOYMENT_ENV=regional ./deploy_infra_services.sh command.
     2. Press Ctrl+c when you see a message similar to the one for the central infrastructure services.
  6. Execute the salt '*' deployutils.get_role_ips 'cassandra' command and confirm that the output displays the IP addresses of all the Salt minions.
  7. Re-run the load services data operation or the health component check that had previously failed.

  Bug Tracking Number: CXU-20815.

• In some cases, high values of round-trip time (RTT) and jitter are displayed in the CSO GUI because of high values reported in the device system log.

  Workaround: None.

  Bug Tracking Number: CXU-21434.

• On an NFX Series CPE device, if you try to upgrade a vSRX gateway router, the upgrade might fail due to a lack of storage space on the VM.

  Workaround:

  Procedure

  Before triggering the upgrade of the vSRX gateway router on an NFX Series device, perform the following steps:

  1. Access the vSRX CLI on the NFX Series device.
  2. Execute the request system storage cleanup command.
  3. Access the JDM CLI on the NFX Series device.
  4. Execute the show virtual-network-function command and note down the name of the vSRX gateway router VM.
  5. Execute the request virtual-network-function gwr-vm-name restart command to reboot the VM, where gwr-vm-name is the name of the vSRX gateway router VM that was obtained in the preceding step.
  6. Wait for the vSRX gateway router VM to successfully reboot.

  Trigger the upgrade of the vSRX gateway router by using the CSO GUI.

  Bug Tracking Number: CXU-21440.

• In some cases, when the infrastructure VMs in the CSO setup are unhealthy and you initiate the upgrade, the upgrade process fails to perform a health check before starting the upgrade.

  Workaround: Recover the infrastructure VMs manually before proceeding with the upgrade.

  Bug Tracking Number: CXU-21536.

• For an MX Series cloud hub device, if you have configured the Internet link type as OAM_and_DATA, the reverse traffic fails to reach the spoke device if you do not configure additional parameters by using the Junos OS CLI on the MX Series device.

  Workaround:

  Procedure

  1. Log in to the MX Series device and access the Junos OS CLI.
  2. Find the next-hop-service outside-service-interface multiservices interface as follows:

     Procedure

     1. Execute the show configuration | display set | grep outside-service-interface command.
     2. In the output of the command, look for the multiservices (ms-) interface corresponding to the service set that CSO created on the device.

        The name of the service set is in the format ssettenant-name_DefaultVPN-tenant-name, where tenant-name is the name of the tenant.

        The following is an example of the command and output:

        show configuration | display set | grep outside-service-interface

        set groups mx-hub-Acme-Acme_DefaultVPN-vpn-routing-config services service-set ssetAcme_DefaultVPN-Acme next-hop-service outside-service-interface ms-1/0/0.4008

        In this example, the tenant name is Acme and the multiservices interface used is ms-1/0/0.4008.

  3. After you determine the correct interface, add the following configuration on the device: set routing-instances WAN_0 interface ms-interface

     where ms-interface is the name of the multiservices interface obtained in the preceding step.

  4. Commit the configuration.

  Bug Tracking Number: CXU-21818.

• In Resource Designer, if you add a VNF that does not require a password and trigger the Add VNF Manager workflow, you are asked to enter a password even though the VNF does not require it.

  Workaround: Even for VNFs that do not require a password, enter a dummy password in Resource Designer when you are creating a VNF package.

  Bug Tracking Number: CXU-21845.

• In a full mesh topology, the simultaneous deletion of LAN segments on all sites is not supported.

  Workaround: Delete LAN segments on one site at a time.

  Bug Tracking Number: CXU-21936.

• When an SRX Series device with factory configuration is activated by using ZTP with a redirect server, the device activation fails because the learned phone home server is deleted during the activation process.

  Workaround: Configure the phone home server IP address on the SRX Series device and retry the ZTP workflow.

  Bug Tracking Number: CXU-22154.

• When you install the CSO Downloader app on MacOS, you might receive an error message indicating that the application cannot be opened because it is from an unidentified developer.

  Workaround: Access the MacOS Security & Privacy settings and allow the CSO Downloader app to be opened and continue with the installation.

  Bug Tracking Number: CXU-22661.

• In small deployments, in rare cases, the DNS lookup fails between microservices, which leads to job failures.

  Workaround:

  Procedure

  1. Access Kibana and check whether the following log is present: Timeout while contacting DNS servers.

  2. • If the log is present:

       Procedure

       1. Log in to the installer VM as root.
       2. Run the ./python.sh recovery/components/reinitialize_pods.py script from the same directory where the CSO package was extracted (untarred).

     • If the log is not present, contact Juniper Networks Technical Support.

  Bug Tracking Number: CXU-23201.

• If you run the script to revert an upgraded CSO Release 4.0.0 setup to CSO Release 3.3.1, the revert operation fails because of an ArangoDB cluster error.

  Workaround: Use the same workaround as CXU-20346.

  Bug Tracking Number: CXU-23338.

• On a CSO setup with secure OAM configured, if you bring up the FortiGate VNF and then apply the license on the VNF, the VNF reboots. However, after rebooting, sometimes the VNF does not come back up.

  Workaround: To ensure that the VNF comes back up, deactivate the VNF and then reactivate it by performing the following steps:

  Procedure

  1. Log in to the JDM CLI of the NFX Series device and access configuration mode.
  2. Deactivate the VNF by executing the deactivate virtual-network-functions Fortinet-oob-2-Firewall command.
  3. Commit the changes by executing the commit command.
  4. Rollback the changes by executing the rollback 1 command
  5. Commit the changes by executing the commit command.
  6. Exit the configuration mode by executing the quit command.
  7. Execute the show virtual-network-functions command and confirm that the status is Running alive, which means that the VNF is up.

  Bug Tracking Number: CXU-23371.

• If one or more VRRs are down, jobs might take a long time to complete, or, in some cases, fail.

  Workaround: Ensure that all VRRs are up before trying the Add Tenant or Add Site workflows.

  Bug Tracking Number: CXU-23710.

• The image upgrade of the vSRX gateway router on NFX Series devices by using the CSO GUI is not supported.

  Workaround: Upgrade the image by using the CLI of the NFX Series device.

  Bug Tracking Number: CXU-23804.

• On an NFX Series device with a Ubuntu VNF instantiated, if you use SSH to log in to the VNF by using the loopback IP address (configured for secure OAM) with port 49154, the connection does not work.

  Workaround:

  Procedure

  1. Use SSH to log in to the vSRX gateway router by using the loopback IP address.
  2. Use SSH to log in to the OAM IP address of the Ubuntu VNF with username root and password passw0rd.
  3. In the Ubuntu VNF, add a route to the IP address of the machine from where you want to log in by using SSH.

     You can now use SSH to log in from the configured machine by using the loopback IP address with port 49154.

  Bug Tracking Number: CXU-23953.

• If all the infrastructure VMs are not up, then the downloading of generated reports fails.

  Workaround: Ensure that all the infrastructure VMs are up and then download the generated reports.

  Bug Tracking Number: CXU-24400.

• On an Ubuntu VNF spawned on an NFX150 device, the ping command to a website address (fully qualified domain name) does not work.

  Workaround:

  Procedure

  1. In Resource Designer, clone the existing ubuntu-fw-NFX150 template for the NFX150 device.
  2. Edit the template and ensure that offloads are disabled for the Left Interface.
  3. Click Next and complete the edit operation.

  Bug Tracking Number: CXU-24441.

• If you are using the GUI installer to install CSO, sometimes the installation page freezes (percentage completion on the VMs does not change) during the installation because of a Rest API timeout.

  Workaround: Reload the CSO installation page in the browser, which will update the status of the installation.

  Bug Tracking Number: CXU-24471.

• When you reboot a device from the Tenant Devices or Devices page, the reboot job fails because connectivity is lost during the reboot.

  Workaround: Check the operational status of the device on the Tenant Devices or Devices page. During the reboot phase, the operational status of the device is Down. After the device is successfully rebooted and connectivity is restored, the operational status of the device changes to Up. You can now trigger operations on the device by using the CSO GUI.

  Bug Tracking Number: CXU-24512.

• If you try to modify a stage-2 configuration template that contains a password configuration, you are asked to reenter the password every time even if you do not want to set the password.

  Workaround: In the Stage-2 Configuration Templates page (Device Template > Stage-2 Config Templates), ensure that the password configuration is moved to the bottom, and do not click the tab corresponding to the password configuration.

  Bug Tracking Number: CXU-24531.

• If you are using the GUI installer to install CSO, sometimes the UI freezes during the installation and no installation progress is seen. However, the installation continues in the backend.

  Workaround: Perform the following tasks:

  Procedure

  1. Reload the installation UI page in the browser.

     If the UI page loads successfully, no further action is needed. If the UI page does not load, proceed to step 2.

  2. Log in to the installer VM as root.
  3. Kill the existing processes triggered by the GUI installer by executing the kill $(sudo lsof -t -i:8080) command.
  4. Navigate to the /root/cso_dl/Contrail_Service_Orchestration_4.0.0/ directory.
  5. Restart the Flask server by executing the bash run_ui.sh command.
  6. After you see the ==== INFO Installer App initialized ===== message on the console, reload the installation UI page in the browser.

  Bug Tracking Number: CXU-24552.

• If all the infrastructure VMs are not up, reports cannot be generated.

  Workaround: Restart the security management monitoring microservice:

  Procedure

  1. Log in to the microservices VM as root.
  2. Run the kubectl get pods | grep csp-secmgt-monitoring command to determine the name of the pod for the security management monitoring microservice.
  3. Restart the pod by executing the kubectl delete pod pod-name command, where pod-name is the name of the microservices pod obtained in the preceding step.
  4. Wait until the pod is in the 1/1 running state.

  You can now retry the report generation.

  Bug Tracking Number: CXU-24560.

• If you try to onboard an NFX150 device with the Hybrid WAN CPE device template, the activation fails after the stage-1 configuration is deployed because connectivity to the device is lost.

  Workaround: You must update the NAT rule configuration to re-enable connectivity between the device and CSO by performing the following steps:

  Procedure

  1. After the stage-1 configuration is deployed, log in to the device.
  2. Log in to the Junos OS CLI and access configuration mode.
  3. Update the source NAT rule configuration on the device to add interface NAT. A sample configuration is shown below.
     set security nat source rule-set deop1-1-lan-wan-ruleset from routing-instance trust

     set security nat source rule-set deop1-1-lan-wan-ruleset to interface ge-1/0/1.0

     set security nat source rule-set deop1-1-lan-wan-ruleset to interface ge-1/0/2.0

     set security nat source rule-set deop1-1-lan-wan-ruleset to interface ge-1/0/9.0

     set security nat source rule-set deop1-1-lan-wan-ruleset rule rule-WAN_0 match source-address 0.0.0.0/0

     set security nat source rule-set deop1-1-lan-wan-ruleset rule rule-WAN_0 then source-nat interface

     set security nat source rule-set deop1-1-lan-wan-ruleset rule rule-WAN_1 match source-address 0.0.0.0/0

     set security nat source rule-set deop1-1-lan-wan-ruleset rule rule-WAN_1 then source-nat interface

  Bug Tracking Number: CXU-24606.

• On a site with network segmentation enabled, if you add a new LAN segment and deploy it, the job is successful. However, the LAN segment's state remains Configured instead of VPN Attached.

  Workaround: Redeploy the LAN segment and the state changes to VPN Attached.

  Bug Tracking Number: CXU-24691.

• The CSO Downloader might not add the default route on the installer VM, if the installer VM is launched in a subnet that is different from the subnet in which the CSO servers are located.

  Workaround: After the installer VM is spawned, add the default route on the installer VM by executing the route add default gw ip-address adapter CLI command, where ip-address is the address of the default gateway and adapter is the name of the network adapter. For example, route add default gw 192.0.2.1 eth0.

  Bug Tracking Number: CXU-24697.

• When you use the CSO GUI installer, VRR behind NAT is not supported for small deployments. However, VRR behind NAT is supported for medium and large deployments using the CSO GUI, or for small, medium, or large deployments using the CLI installer.

  Workaround: If you want to use the VRR behind NAT feature in a small deployment, use the CLI-based installer and not the GUI installer.

  Bug Tracking Number: CXU-24699.

• If a user with an OpCo Admin role clones a device template and a tenant of that OpCo creates a site using the cloned device template, the site creation operation fails.

  Workaround: Ensure that device templates are cloned or modified only by users with the SP Admin role.

  Bug Tracking Number: CXU-24799.

• If the CSO GUI installer is used in the custom install mode and after CSO is successfully installed, you click the Launch CSO Admin Portal button, the installer tries to invoke Administration Portal by using the VRR IP address instead of the IP address of the central microservices VM.

  Workaround: Use the IP address of the central microservices VM to launch Administration Portal in your browser.

  Bug Tracking Number: CXU-24907.

• The Help (?) menu for the Administration Portal and Customer Portal is empty and does not display any links.

  Workaround: Use the following links to access the Administration Portal and Customer Portal Help Centers:

  • Administration Portal: https://www.juniper.net/documentation/en_US/cso4.0/help/information-products/pathway-pages/admin-portal-index.html

  • Customer Portal: https://www.juniper.net/documentation/en_US/cso4.0/help/information-products/pathway-pages/customer-portal/cp-index.html

  You can also use the More... hyperlink on a page to access the online help content for that page.

  Bug Tracking Number: CXU-24941.