Known Issues

This section lists known issues in Juniper Networks CSO Release 4.0.1.

AWS Spoke

• The AWS device activation process takes up to 30 minutes. If the process does not complete in 30 minutes, a timeout might occur and you must retry the process. You do not need to download the cloud formation template again.

  Procedure

  To retry the process:

  1. Log in to Customer Portal.
  2. Access the Activate Device page, enter the activation code, and click Next.
  3. After the CREATE_COMPLETE message is displayed on the AWS server, click Next on the Activate Device page to proceed with device activation.

  Bug Tracking Number: CXU-19102.

CSO HA

• In a CSO HA environment, two RabbitMQ nodes are clustered together, but the third RabbitMQ node does not join the cluster. This might occur just after the initial installation, if a virtual machine reboots, or if a virtual machine is powered off and then powered on.

  Workaround: Do the following:

  Procedure

  1. Log in to the installer VM.
  2. Navigate to the current deployment directory for CSO—for example, /root/Contrail_Service_Orchestration_4.0.1/.
  3. Execute the ./recovery.sh command.
  4. Specify the option to recover RabbitMQ and press Enter.
  5. In the RabbitMQ dashboards for the central and regional microservices VMs, confirm that all the available infrastructure nodes are present in the cluster.

  Bug Tracking Number: CXU-12107

• In an HA setup, the time configured for the CAN VMs might not be synchronized with the time configured for the other VMs in the setup. This can cause issues in the throughput graphs.

  Procedure

  Workaround:

  1. Log in to can-vm1 as the root user.
  2. Modify the /etc/ntp.conf file to point to the desired NTP server.
  3. Restart the NTP process.

  After the NTP process restarts successfully, can-vm2 and can-vm3 automatically resynchronize their times with can-vm1.

  Bug Tracking Number: CXU-15681.

• When a high availability (HA) setup comes back up after a power outage, MariaDB instances do not come back up on the VMs.

  Workaround:

  Procedure

  Perform the following steps to recover the MariaDB instances:

  1. Log in to the installer VM.
  2. Navigate to the current deployment directory for CSO; for example, /root/Contrail_Service_Orchestration_4.0.1/.
  3. Execute the sed -i "s@/var/lib/mysql/grastate.dat@/mnt/data/mysql/grastate.dat@g" recovery/components/recover_mariadb.py command
  4. Execute the ./recovery.sh command.
  5. Specify the option to recover MariaDB and press Enter.

  Bug Tracking Number: CXU-20260.

• In some cases, when power fails, the ArangoDB cluster does not form.

  Workaround:

  Procedure

  1. Log in to the centralinfravm3 VM.
  2. Execute the following commands:
     service arangodb3.cluster stop

     cd /var/lib/arangodb3 && mv setup.json setup.json.old
  3. Log in to the centralinfravm2 VM.
  4. Execute the following commands:
     service arangodb3.cluster stop

     cd /var/lib/arangodb3 && mv setup.json setup.json.old
  5. Log in to the centralinfravm1 VM.
  6. Execute the following commands:
     service arangodb3.cluster stop

     cd /var/lib/arangodb3 && mv setup.json setup.json.old
  7. On the centralinfravm1 VM, execute the service arangodb3.cluster start command and wait for 20 seconds for the command to finish executing.
  8. On the centralinfravm2 VM, execute the service arangodb3.cluster start command and wait for 20 seconds for the command to finish executing.
  9. On the centralinfravm3 VM, execute the service arangodb3.cluster start command and wait for 20 seconds for the command to finish executing.

  Bug Tracking Number: CXU-20346.

• In a HA setup, if you shut down all the CSO servers, after the servers are restarted successfully, MariaDB and ArangoDB fail to form their respective clusters.

  Workaround:

  Procedure

  1. Perform a clean reboot of the central infrastructure VMs.
  2. After the VMs have rebooted successfully, check the cluster health on the HAproxy page (http://central-ip-address:1936, where central-IP-address is the IP address of the VM that hosts the microservices for the central POP).
  3. If the MariaDB and ArangoDB clusters are still down, you can recover the clusters by performing the following procedures:

     • To recover the MariaDB cluster, perform the following steps:

       Procedure

       1. On the centralinfravm1 VM, execute the service mysql stop command.
       2. On the centralinfravm2 VM, execute the service mysql stop command.
       3. On the centralinfravm3 VM, execute the service mysql stop command.
       4. On all three central infrastructure VMs, verify that the service has stopped executing the service mysql status command.
       5. On the centralinfravm1 VM, start the service by executing the service mysql start command.
       6. On the centralinfravm2 VM, start the service by executing the service mysql start command.
       7. On the centralinfravm3 VM, start the service by executing the service mysql start command.
       8. On all three central infrastructure VMs, verify that the service has started executing the service mysql status command.

     • To recover the ArangoDB cluster, perform the following steps:

       Procedure

       1. On the centralinfravm1 VM, execute the service arangodb3.cluster stop command.
       2. On the centralinfravm2 VM, execute the service arangodb3.cluster stop command.
       3. On the centralinfravm3 VM, execute the service arangodb3.cluster stop command.
       4. On all three central infrastructure VMs, verify that the service has stopped executing the ps -aef|grep arangodb command.
       5. On the centralinfravm1 VM, start the service by executing the service arangodb3.cluster start command.
       6. On the centralinfravm2 VM, start the service by executing the service arangodb3.cluster start command.
       7. On the centralinfravm3 VM, start the service by executing the service arangodb3.cluster start command.
       8. On all three central infrastructure VMs, verify that the service has started executing the ps -aef|grep arangodb command.

  Bug Tracking Number: CXU-21819.

• In a HA setup, if you onboard devices and deploy policies on the devices and if one of the policy deployments is in progress when a microservices or infrastructure node goes down, the deployment job is stuck in the In Progress state for about 90 minutes (the default timeout value), and you cannot perform deploy operations for the tenant for about 90 minutes.

  Workaround: Wait for the job to fail and then redeploy the policy.

  Bug Tracking Number: CXU-21922.

• If an infrastructure node goes down in a HA setup in which all nodes were previously up, and you create a firewall policy and try to deploy the policy, the deployment job is stuck in the in-progress state and a Redis timeout error is displayed in the job log.

  Workaround:

  Procedure

  1. Wait for approximately 90 minutes for the job to fail.
  2. Bring up the infrastructure node that was down.
  3. Redeploy the firewall policy.

  Bug Tracking Number: CXU-24559.

• While you are upgrading CSO (Production Environment with HA) from Release 3.3.1 to Release 4.0.1, the upgrade fails after a snapshot is taken because the regional Kubernetes node is in the Not Ready status.

  Workaround: Restart the Docker service.

  Procedure

  1. On the regional K8 master node, run the following commands:

     • service docker stop

     • rm -rf /var/lib/docker/*

     • service docker start

  2. Run the ./upgrade.sh script.

  Bug Tracking Number: CXU-25625.

• In CSO Release 4.0.1, the site upgrade fails for sites with vSRX as a CPE device.

  Workaround: Before upgrading the site, manually execute the request system storage cleanup command on the vSRX CLI.

  Bug Tracking Number: CXU-25713.

• You cannot access the Administration Portal login page if the flannel network subnet is changed.

  Workaround:

  1. Log in to central microservices VMs.

  2. In all central microservices VMs, run the following commands in parallel:

     service flanneld stop

     service docker stop

     service flanneld start

     sleep 10

     service docker start

     After executing these commands, wait for 10 minutes.

  3. On any one of the central microservices VM, run the following command to delete all pods:

     kubectl delete pods --all --force --grace-period=0 -n central

  Bug Tracking Number: CXU-23736.

SD-WAN

• On the Site SLA Performance page, applications with different SLA scores are plotted at the same coordinate on the x-axis.

  Workaround: None.

  Bug Tracking Number: CXU-19768.

• When all local breakout links are down, site to Internet traffic fails even though there is an active overlay to the hub.

  Workaround: None.

  Bug Tracking Number: CXU-19807

• In a Cloud hub multihoming topology, after a link switch, the GRE tunnel links on the secondary hub might be displayed in red in the CSO GUI, even though the GRE tunnels are up.

  Workaround: Wait for approximately 10 minutes and the links are displayed in green, indicating that the GRE tunnels are up.

  Bug Tracking Number: CXU-20550.

• If the Internet breakout WAN link of the cloud hub is not used for provisioning the overlay tunnel by at least one spoke site in a tenant, then traffic from sites to the Internet is dropped.

  Workaround: Ensure that you configure a firewall policy to allow traffic from security zone trust-tenant-name to zone untrust-wan-link, where tenant-name is the name of the tenant and wan-link is the name of the Internet breakout WAN link.

• Bug Tracking Number: CXU-21291.

• On the SD-WAN Events page, for link switch events, if you mouse over the Reason field, the values displayed for the SLA metrics are the ones that are recorded when the system logs are sent from the device and not the values for which the SLA violation was detected.

  Workaround: None.

  Bug Tracking Number: CXU-21461.

• In a hub-and-spoke topology with multitenancy enabled, when a spoke site is configured with two MPLS and two Internet links with MPLS selected as the default, traffic from the hub to the spoke site takes the same path instead of taking the path (link) on which the traffic was received by the hub (incoming WAN link). However, there is no traffic loss.

  Workaround: Remove the static route with the next hop and replace it with a static route with the qualified next hop.

  Bug Tracking Number: CXU-23197.

• If a WAN link on a CPE device goes down, the WAN tab of the Site-Name page (in Administration Portal) displays the corresponding link metrics as N/A.

  Workaround: None.

  Bug Tracking Number: CXU-23996.

• If a tenant has a real-time-optimized site, link switch events (on the Monitor page) might display the same WAN link for both source and destination tunnels.

  Workaround: None.

  Bug Tracking Number: CXU-24154.

• When you delete an SD-WAN policy, the custom application group associated with the SD-WAN is not deleted.

  Workaround: Log in to the device, and delete the custom application group.

  Bug Tracking Number: CXU-25657.

• If you delete a cloud hub that is created in Release 3.3.1, CSO does not delete the stage-2 configuration.

  Workaround: You must manually delete the stage-2 configuration from the device.

  Bug Tracking Number: CXU-25764

Security Management

• On the Active Database page in Customer Portal, the wrong installed device count is displayed. The count displayed is for all tenants and not for a specific tenant.

  Workaround: None.

  Bug Tracking Number: CXU-20531.

• If a cloud hub is used by two tenants, one with public key infrastructure (PKI) authentication enabled and other with preshared key (PSK) authentication enabled, the commit configuration operation fails. This is because only one IKE gateway can point to one policy and if you define a policy with a certificate then the preshared key does not work.

  Workaround: Ensure that the tenants sharing a cloud hub use the same type of authentication (either PKI or PSK) as the cloud hub device.

  Bug Tracking Number: CXU-23107.

• If UTM Web-filtering categories are installed manually (by using the request system security UTM web-filtering category install command from the CLI) on an NFX150 device, the intent-based firewall policy deployment from CSO fails.

  Workaround: Uninstall the UTM Web-filtering category that you installed manually by executing the request security utm web-filtering category uninstall command on the NFX150 device and then deploy the firewall policy.

  Bug Tracking Number: CXU-23927.

• On the Identity Management page, if you click Download JIMS, the Juniper Identity Management Service (JIMS) software is downloaded in HTML format.

  Workaround: Download the JIMS software from the Download Software page.

  Bug Tracking Number: CXU-24278.

• If SSL proxy is configured on a dual CPE device and if the traffic path is changed from one node to another node, the following issue occurs:

  • For cacheable applications, if there is no cache entry the first session might fail to establish.

  • For non-cacheable applications, the traffic flow is impacted.

  Workaround: None.

  Bug Tracking Number: CXU-25526.

• On a gateway router, the application identification license and the application signature that are installed are lost.

  Workaround:

  • To manually install the application identification license, run the request system license add license_file command.

  • To install application signatures, see Installing Signatures in the CSO User Guide.

  Bug Tracking Number: CXU-25641.

• The UTM policy configuration is not deployed on an SD-WAN site with the SRX device model SRX345-DUAL-AC.

  Workaround:

  1. Add the SRX345-DUAL-AC device model to the schema file.

     Note In the schema-svc docker, the schema file is available at /opt/csp-schema-data/*configuration.json.

  2. Restart the pod.

  Bug Tracking Number: CXU-25706.

• On a Dual CPE device, the deployment of the AllSite_AUTONAT_Policy policy fails.

  Workaround: Redeploy the pending AllSite_AUTONAT_Policy policy. For more information about deploying a policy, see Deploying Policies in the CSO User Guide.

  Bug Tracking Number: CXU-25855.

Site and Tenant Workflow

• The tenant delete operation fails when CSO is installed with an external Keystone.

  Workaround: You must manually delete the tenant from the Contrail OpenStack user interface.

  Bug Tracking Number: CXU-9070

• If you try to activate a branch SRX Series device with the factory-default configuration, the stage-1 configuration commit might fail when there are active DHCP server bindings on the device. This is because of the default DHCP server settings present in factory-default configuration.

  Workaround: When you are pre-staging the CPE device for activation, remove the DHCP server-related configuration from the device by executing the following commands on the Junos OS CLI:

  set system services dhcp-local-server group jdhcp-group interface fxp0.0

  set system services dhcp-local-server group jdhcp-group interface irb.0

  Bug Tracking Number: [CXU-13446]

• In some cases, if automatic license installation is enabled in the device profile, after ZTP is complete, the license might not be installed on the CPE device even though license key is configured successfully.

  Workaround: Reinstall the license on the CPE device by using the Licenses page on the Administration Portal.

  Bug Tracking Number: PR1350302.

• For a tenant, LAN segments with overlapping IP prefixes across sites are not supported.

  Workaround: Create LAN segments with unique IP prefixes across sites for the tenant.

  Bug Tracking Number: CXU-20494.

• When the primary and backup interfaces of the CPE device uses the same WAN interface of the hub, the backup underlay might be used for Internet or site-to-site traffic even though the primary links are available.

  Workaround: Ensure that you connect the WAN links of each CPE device to unique WAN links of the hub.

  Bug Tracking Number: CXU-20564.

• After you configure a site, you cannot modify the configuration either before or after activation.

  Workaround: None.

  Bug Tracking Number: CXU-21165.

• If you initiate the RMA workflow on an NFX Series device that was successfully onboarded and provisioned with stage-2 templates, the device RMA operation might get stuck in the device activation stage if the stage-2 configuration templates have interdependencies.

  Workaround: Ensure that the stage-2 templates that are deployed on the device do not have interdependencies before initiating the device RMA workflow.

  Bug Tracking Number: CXU-21464.

• On the Monitor > Overview page, if you click a site indicating that a major alarm was triggered (site icon color turns orange), and in the subsequent popup, click the link for major alarms in the Alerts & Alarms section, you are taken to the Alarms page. However, no alarm for the device is displayed.

  Workaround: None.

  Bug Tracking Number: CXU-21828.

• If a tenant is deleted and a different tenant is added with the same name as the previously deleted tenant, the ZTP of the NFX Series spoke device fails during the VRR reconfiguration.

  Workaround:

  Procedure

  1. Delete the tenant.
  2. Add the tenant with a name that is different from that of the deleted tenant.
  3. Retry the ZTP workflow.

  Bug Tracking Number: CXU-24260.

• After you upgrade a site with an NFX250 device, the monitoring page does not display any data. This is because the telemetry agent is uninstalled during the site upgrade.

  Workaround: Recall the NFX250 device and perform ZTP. The telemetry agent is installed and the monitoring page begins to display the data.

  Bug Tracking Number: CXU-19455.

• On the Configure Site page, the values that you specify for the time zone and the IP address of the NTP server are not being pushed to the device.

  Workaround: Configure the NTP server IP address and the time zone on the device, manually:

  1. Log in to the device.

  2. In the configuration mode, run the following commands:

     • set system ntp server IP address

     • set system time-zone time zone

  3. Commit the changes

  Bug Tracking Number: CXU-23971.

• On an NFX250 device, if you disable (detach) a failed service successfully and then try to delete the site, the site is not deleted.

  Workaround: None.

  Bug Tracking Number: CXU-24355.

• When you try to activate a site with an SRX Series device, ZTP might fail with an error during the installation of the default trusted certificates.

  Workaround: Retry the failed job after some time.

  Bug Tracking Number: CXU-24487.

• If you try to activate a site with an MPLS link by using DHCP, the default route pointing to the MPLS gateway is added to the hub device, which results in Internet traffic from the hub taking the MPLS link.

  Workaround: None.

  Bug Tracking Number: CXU-24666.

• On the Import Sites page, the operation to import multiple sites by using a JSON file fails.

  Workaround: Use the Sites page to create sites.

  Bug Tracking Number: CXU-24730.

• If you trigger the tenant creation workflow, the tenant might be displayed in the CSO GUI even before the job is completed. If you then try to trigger workflows for that tenant, the subsequent jobs fail because the tenant creation job is not completed.

  Workaround: Wait for the tenant creation job to complete successfully before triggering any workflows for the tenant.

  Bug Tracking Number: CXU-24783.

• The Configure Site operation for a cloud spoke site fails.

  Workaround: None.

  Bug Tracking Number: CXU-24795.

• The site upgrade for a dual CPE site fails because CSO does not support image upgrade.

  Workaround: You must manually upgrade the image.

  Bug Tracking Number: CXU-24823.

• The tenant name cannot be the same across different operating companies.

  Workaround: None.

  Bug Tracking Number: CXU-25225.

• For an Application Quality of Experience (AppQoE) tenant, an SLA profile with only the Throughput parameter is not supported.

  Workaround: Specify the other SLA parameters (Latency, Packet Loss, Jitter, and so on) along with the Throughput parameter.

  Bug Tracking Number: CXU-25378.

• After you upgrade CSO to Release 4.0.1, you cannot delete tenants that are created in CSO Release 3.3.1.

  Workaround: You can delete the tenant after you delete sites and departments that are associated with the tenant.

  Bug Tracking Number: CXU-25716.

• If network segmentation is enabled for a tenant, when you add and deploy a new LAN segment, the status of the LAN segment is displayed as Configured instead of VPN Attached.

  Workaround: Redeploy the LAN segment. For more information about deploying a LAN segment, see Deploying a LAN Segment in the CSO User Guide.

  Bug Tracking Number: CXU-25734.

• The Configure Site operation fails if you import a cloud hub with a name that is different from that of other tenants.

  Workaround: While you are importing a cloud hub, specify the same name that is used while onboarding a cloud hub for a global service provider.

  Bug Tracking Number: CXU-25740.

• You cannot configure a site with dual CPE devices if WAN links are used exclusively for local breakout traffic.

  Workaround: While you are creating a site and enabling the link for local breakout, instead of selecting the Use only for breakout traffic option, select Use for breakout & WAN traffic. Also, while you are configuring a site ensure that the WAN link is connected to a hub.

  Bug Tracking Number: CXU-25776.

• For some sites, the Configure Site operation fails and the following error message is displayed:

  Policy is out of sync between RE and PFE fpc0. Please resync before commit.

  Workaround:

  1. Log in to the cloud hub associated with the site.

  2. Execute the following command to synchronize the policies between the Packet Forwarding Engine and Routine Engine:

     request security policies resync

  3. From the CSO GUI, delete the site.

  4. Re-create the site.

  Bug Tracking Number: CXU-25808.

General

• If you create VNF instances in the Contrail cloud by using Heat Version 2.0 APIs, a timeout error occurs after 120 instances are created.

  Workaround: Contact Juniper Networks Technical Support.

  Bug Tracking Number: CXU-15033

• When you upgrade the gateway router by using the CSO GUI, after the upgrade completes and the gateway router reboots, the gateway router configuration reverts to the base configuration and loses the IPsec configuration added during Zero Touch Provisioning (ZTP).

  Workaround: Before you upgrade the gateway router by using the CSO GUI, ensure that you do the following:

  Procedure

  1. Log in to the Juniper Device Manager (JDM) CLI of the NFX Series device.
  2. Execute the virsh list command to obtain the name of the gateway router (GWR_NAME).
  3. Execute the request virtual-network-functions GWR_NAME restart command, where GWR_NAME is the name of the gateway router obtained in the preceding step.
  4. Wait a few minutes for the gateway router to come back up.
  5. Log out of the JDM CLI.
  6. Proceed with the upgrade of the gateway router by using the CSO GUI.

  Bug Tracking Number: CXU-11823.

• CSO might not come up after a power failure.

  Workaround:

  Procedure

  1. Log in to the installer VM.
  2. Navigate to the /root/Contrail_Service_Orchestration_4.0.1/ directory.
  3. Run the reinitialize_pods.py script as follows:
     ./python.sh recovery/components/reinitialize_pods.py
  4. SSH to the VRR by using the VRR IP address to check if you are able to access the VRR.

     If there is an error in connecting (port 22: Connection refused), then you must recover the VRR by following step 5 through 21.

  5. Log in to physical server hosting the VRR.
  6. Execute the virsh destroy vrr command to destroy the VRR.

     Warning Do not execute the virsh undefine vrr command because doing so will cause the VRR configuration to be lost and the configuration cannot be recovered.

  7. Delete the VRR image that is located in the /root/ubuntu_vm/vrr/vrr-15.1R6.7.qcow2 directory.
  8. Copy the fresh VRR image from the /root/disks/vrr-15.1R6.7.qcow2 directory to the /root/ubuntu_vm/vrr/vrr-15.1R6.7.qcow2 directory.
  9. Execute the virsh start vrr command and wait for approximately 5 minutes for the command to finish executing.
  10. Execute the virsh list –all command to check if the VRR is running or not.

      If the VRR is not running, check that the image that was copied was the uncorrupted image and re-try the steps from step 7.

  11. If the VRR is running, navigate to the /root/ubuntu_vm/vrr/ directory.
  12. Run the ./vrr.exp command to push the base configuration to the VRR.
  13. Check if the VRR is reachable from the regional microservices VM. If the VRR is reachable, proceed to step 14. If the VRR is not reachable:

      Procedure

      1. Log in to the VRR.
      2. Check if the base configuration was pushed properly:

         • If the base configuration was pushed properly, re-check if the VRR is reachable from the regional microservices VM. If the VRR is reachable, proceed to step 14.

         • If the base configuration was not pushed properly:

           Procedure

           1. If the base configuration was not pushed properly, add the necessary routes to reach CSO.
           2. Re-check if the VRR is reachable from the regional microservices VM. If the VRR is reachable, proceed to 14.
  14. Import the POP by using the URL https://central-ms-ip:443/tssm/import-pop, where central-ms-ip is the IP address of the central microservices VM.
  15. Use POSTMAN to import the VRR.

      Note Do not import the VRR until the VRR is reachable from the regional microservices VM.

      The following is JSON format for the VRR. (In the JSON below, <vrr-ip-address> is the IP address of the VRR and <vrr-password> is the password that was configured for the VRR.

      {                "input": {                                "job_name_prefix": "ImportPop",                                "pop": [{                                                "dc_name": "regional",                                                "device": [{                                                                "name": "vrr-<vrr-ip-address>",                                                                "family": "VRR",                                                                "device_ip": "<vrr-ip-address>",                                                                "assigned_device_profile": "VRR_Advanced_SDWAN_option_1",                                                                "authentication": {                                                                                "password_based": {                                                                                                "username": "root",                                                                                                "password": "<vrr-password>"                                                                                }                                                                },                                                                "management_state": "managed",                                                                "pnf_package": "null"                                                }],                                                "name": "regional"                                }]                }}

  16. Verify whether the VRR configuration is imported properly:

      Procedure

      1. Log in to the CSO Administration Portal.
      2. Click Resources > POPs > Import POPs > Import History and confirm that the ImportPop job is running and that it has completed successfully.
  17. On the Tenants page, add a tenant named recovery.
  18. After the tenant is successfully created, log in to the VRR and access the Junos OS CLI.
  19. Execute the show configuration|display set and verify that the tenant configuration (for the previously-configured tenants) is recovered.
  20. Execute the show bgp summary and check that the BGP status to the hub and spokes are Established.
  21. If the status is Not Established, add the routes for the OAM traffic of the hub and spokes to the VRR and recheck the status.

  Bug Tracking Number: CXU-16530

• The provisioning of CPE devices fails if all VRRs within a redundancy group are unavailable.

  Workaround: Recover the VRR that is down and retry the provisioning job.

  Bug Tracking Number: CXU-19063

• The CSO health check displays the following error message: ERROR: ONE OR MORE KUBE-SYSTEM PODS ARE NOT RUNNING

  Workaround:

  Procedure

  1. Log in to the central microservices VM.
  2. Execute the kubectl get pods –namespace=kube-system command.
  3. If the kube-proxy process is not in the Running state, execute the kubectl apply –f /etc/kubernetes/manifests/kube-proxy.yaml command.

     Bug Tracking Number: CXU-20275.

• After the upgrade, the health check on the standalone Contrail Analytics Node (CAN) fails.

  Workaround:

  Procedure

  1. Log in to the CAN VM.
  2. Execute the docker exec analyticsdb service contrail-database-nodemgr restart command.
  3. Execute the docker exec analyticsdb service cassandra restart command.

  Bug Tracking Number: CXU-20470.

• The class-of-service scheduler configuration does not take effect on the CPE device.

  Workaround:

  Procedure

  1. Log in to the CPE device and access the Junos OS CLI.
  2. Enable the scheduler map on each physical interface manually by executing the following commands:
     set class-of-service interfaces interface-name unit * scheduler-map scheduler-map-name

     set interfaces interface-name per-unit-scheduler

     Where interface-name is the name of the physical interface (for example, ge-0/0/4), and scheduler-map-name is the name of the scheduler map.

  3. Commit the configuration on the CPE device.

  Bug Tracking Number: CXU-20708.

• The load services data operation or health check of the infrastructure components might fail if the data in the Salt server cache is lost because of an error.

  Workaround: If you encounter a Salt server-related error, do the following:

  Procedure

  1. Log in to the installer VM.
  2. Execute the salt '*' deployutils.get_role_ips 'cassandra' command to confirm whether one or more Salt minions have lost the cache.

     • If the output returns the IP address for all the Salt minions, this means that the Salt server cache is fine; proceed to step 7.

     • If the IP address for some minions is not present in the output, this means that the Salt server has lost its cache for those minions and must be rebuilt as explained from step 3.

  3. Navigate to the current deployment directory for CSO; for example, /root/Contrail_Service_Orchestration_4.0.1/.
  4. Redeploy the central infrastructure services (up to the NTP step):

     Procedure

     1. Execute the DEPLOYMENT_ENV=central ./deploy_infra_services.sh command.
     2. Press Ctrl+c when you see the following message on the console:

        2018-04-10 17:17:03 INFO utils.core Deploying roles set(['ntp']) to servers ['csp-central-msvm', 'csp-contrailanalytics-1', 'csp-central-k8mastervm', 'csp-central-infravm']

  5. Redeploy the regional infrastructure services (up to the NTP step):

     Procedure

     1. Execute the DEPLOYMENT_ENV=regional ./deploy_infra_services.sh command.
     2. Press Ctrl+c when you see a message similar to the one for the central infrastructure services.
  6. Execute the salt '*' deployutils.get_role_ips 'cassandra' command and confirm that the output displays the IP addresses of all the Salt minions.
  7. Re-run the load services data operation or the health component check that had previously failed.

  Bug Tracking Number: CXU-20815.

• In some cases, high values of round-trip time (RTT) and jitter are displayed in the CSO GUI because of high values reported in the device system log.

  Workaround: None.

  Bug Tracking Number: CXU-21434.

• On an NFX Series CPE device, if you try to upgrade a vSRX gateway router, the upgrade might fail due to a lack of storage space on the VM.

  Workaround:

  Procedure

  Before triggering the upgrade of the vSRX gateway router on an NFX Series device, perform the following steps:

  1. Access the vSRX CLI on the NFX Series device.
  2. Execute the request system storage cleanup command.
  3. Access the JDM CLI on the NFX Series device.
  4. Execute the show virtual-network-function command and note down the name of the vSRX gateway router VM.
  5. Execute the request virtual-network-function gwr-vm-name restart command to reboot the VM, where gwr-vm-name is the name of the vSRX gateway router VM that was obtained in the preceding step.
  6. Wait for the vSRX gateway router VM to successfully reboot.

  Trigger the upgrade of the vSRX gateway router by using the CSO GUI.

  Bug Tracking Number: CXU-21440.

• In some cases, when the infrastructure VMs in the CSO setup are unhealthy and you initiate the upgrade, the upgrade process fails to perform a health check before starting the upgrade.

  Workaround: Recover the infrastructure VMs manually before proceeding with the upgrade.

  Bug Tracking Number: CXU-21536.

• For an MX Series cloud hub device, if you have configured the Internet link type as OAM_and_DATA, the reverse traffic fails to reach the spoke device if you do not configure additional parameters by using the Junos OS CLI on the MX Series device.

  Workaround:

  Procedure

  1. Log in to the MX Series device and access the Junos OS CLI.
  2. Find the next-hop-service outside-service-interface multiservices interface as follows:

     Procedure

     1. Execute the show configuration | display set | grep outside-service-interface command.
     2. In the output of the command, look for the multiservices (ms-) interface corresponding to the service set that CSO created on the device.

        The name of the service set is in the format ssettenant-name_DefaultVPN-tenant-name, where tenant-name is the name of the tenant.

        The following is an example of the command and output:

        show configuration | display set | grep outside-service-interface

        set groups mx-hub-Acme-Acme_DefaultVPN-vpn-routing-config services service-set ssetAcme_DefaultVPN-Acme next-hop-service outside-service-interface ms-1/0/0.4008

        In this example, the tenant name is Acme and the multiservices interface used is ms-1/0/0.4008.

  3. After you determine the correct interface, add the following configuration on the device: set routing-instances WAN_0 interface ms-interface

     where ms-interface is the name of the multiservices interface obtained in the preceding step.

  4. Commit the configuration.

  Bug Tracking Number: CXU-21818.

• In Resource Designer, if you add a VNF that does not require a password and trigger the Add VNF Manager workflow, you are asked to enter a password even though the VNF does not require it.

  Workaround: Even for VNFs that do not require a password, enter a dummy password in Resource Designer when you are creating a VNF package.

  Bug Tracking Number: CXU-21845.

• In a full mesh topology, the simultaneous deletion of LAN segments on all sites is not supported.

  Workaround: Delete LAN segments on one site at a time.

  Bug Tracking Number: CXU-21936.

• When an SRX Series device with factory configuration is activated by using ZTP with a redirect server, the device activation fails because the learned phone home server is deleted during the activation process.

  Workaround: Configure the phone home server IP address on the SRX Series device and retry the ZTP workflow.

  Bug Tracking Number: CXU-22154.

• When you install the CSO Downloader app on MacOS, you might receive an error message indicating that the application cannot be opened because it is from an unidentified developer.

  Workaround: Access the MacOS Security & Privacy settings and allow the CSO Downloader app to be opened and continue with the installation.

  Bug Tracking Number: CXU-22661.

• In small deployments, in rare cases, the DNS lookup fails between microservices, which leads to job failures.

  Workaround:

  Procedure

  1. Access Kibana and check whether the following log is present: Timeout while contacting DNS servers.

  2. • If the log is present:

       Procedure

       1. Log in to the installer VM as root.
       2. Run the ./python.sh recovery/components/reinitialize_pods.py script from the same directory where the CSO package was extracted (untarred).

     • If the log is not present, contact Juniper Networks Technical Support.

  Bug Tracking Number: CXU-23201.

• If you run the script to revert an upgraded CSO Release 4.0.0 setup to CSO Release 3.3.1, the revert operation fails because of an ArangoDB cluster error.

  Workaround: Use the same workaround as CXU-20346.

  Bug Tracking Number: CXU-23338.

• On a CSO setup with secure OAM configured, if you bring up the FortiGate VNF and then apply the license on the VNF, the VNF reboots. However, after rebooting, sometimes the VNF does not come back up.

  Workaround: To ensure that the VNF comes back up, deactivate the VNF and then reactivate it by performing the following steps:

  Procedure

  1. Log in to the JDM CLI of the NFX Series device and access configuration mode.
  2. Deactivate the VNF by executing the deactivate virtual-network-functions Fortinet-oob-2-Firewall command.
  3. Commit the changes by executing the commit command.
  4. Rollback the changes by executing the rollback 1 command
  5. Commit the changes by executing the commit command.
  6. Exit the configuration mode by executing the quit command.
  7. Execute the show virtual-network-functions command and confirm that the status is Running alive, which means that the VNF is up.

  Bug Tracking Number: CXU-23371.

• If one or more VRRs are down, jobs might take a long time to complete, or, in some cases, fail.

  Workaround: Ensure that all VRRs are up before trying the Add Tenant or Add Site workflows.

  Bug Tracking Number: CXU-23710.

• The image upgrade of the vSRX gateway router on NFX Series devices by using the CSO GUI is not supported.

  Workaround: Upgrade the image by using the CLI of the NFX Series device.

  Bug Tracking Number: CXU-23804.

• On an NFX Series device with a Ubuntu VNF instantiated, if you use SSH to do log in to the VNF by using the loopback IP address (configured for secure OAM) with port 49154, the connection does not work.

  Workaround:

  Procedure

  1. Use SSH to log in to the vSRX gateway router by using the loopback IP address.
  2. Use SSH to log in to the OAM IP address of the Ubuntu VNF with username root and password passw0rd.
  3. In the Ubuntu VNF, add a route to the IP address of the machine from where you want to log in by using SSH.

     You can now use SSH to log in from the configured machine by using the loopback IP address with port 49154.

  Bug Tracking Number: CXU-23953.

• On an Ubuntu VNF spawned on an NFX150 device, the ping command to a website address (fully qualified domain name) does not work.

  Workaround:

  Procedure

  1. In Resource Designer, clone the existing ubuntu-fw-NFX150 template for the NFX150 device.
  2. Edit the template and ensure that offloads are disabled for the Left Interface.
  3. Click Next and complete the edit operation.

  Bug Tracking Number: CXU-24441.

• If you are using the GUI installer to install CSO, sometimes the installation page freezes (percentage completion on the VMs does not change) during the installation because of a Rest API timeout.

  Workaround: Reload the CSO installation page in the browser, which will update the status of the installation.

  Bug Tracking Number: CXU-24471.

• When you reboot a device from the Tenant Devices or Devices pages, the reboot job fails because the connectivity is lost during the reboot.

  Workaround: Check the operational status of the device on the Tenant Devices or Devices page. During the reboot phase, the operational status of the device is Down. After the device is successfully rebooted and connectivity is restored, the operational status of the device changes to Up. You can now trigger operations on the device by using the CSO GUI.

  Bug Tracking Number: CXU-24512.

• If a user with an OpCo Admin role clones a device template and a tenant of that OpCo creates a site using the cloned device template, the site creation operation fails.

  Workaround: Ensure that device templates are cloned or modified only by users with the SP Admin role.

  Bug Tracking Number: CXU-24799

• If you are using the GUI installer to install CSO, sometimes the UI freezes during the installation and no installation progress is seen. However, the installation continues in the backend.

  Workaround: Perform the following tasks:

  Procedure

  1. Reload the installation UI page in the browser.

     If the UI page loads successfully, no further action is needed. If the UI page does not load, proceed to step 2.

  2. Log in to the installer VM as root.
  3. Kill the existing processes triggered by the GUI installer by executing the kill $(sudo lsof -t -i:8080) command.
  4. Navigate to the /root/cso_dl/Contrail_Service_Orchestration_4.0.1/ directory.
  5. Restart the Flask server by executing the bash run_ui.sh command.
  6. After you see the ==== INFO Installer App initialized ===== message on the console, reload the installation UI page in the browser.

  Bug Tracking Number: CXU-24552.

• If all the infrastructure VMs are not up, reports cannot be generated.

  Workaround: Restart the security management monitoring microservice:

  Procedure

  1. Log in to the microservices VM as root.
  2. Run the kubectl get pods | grep csp-secmgt-monitoring command to determine the name of the pod for the security management monitoring microservice.
  3. Restart the pod by executing the kubectl delete pod pod-name command, where pod-name is the name of the microservices pod obtained in the preceding step.
  4. Wait until the pod is in the 1/1 running state.

  You can now retry the report generation.

  Bug Tracking Number: CXU-24560.

• If you try to onboard an NFX150 device with the Hybrid WAN CPE device template, the activation fails after the stage-1 configuration is deployed because connectivity to the device is lost.

  Workaround: You must update the NAT rule configuration to reenable connectivity between the device and CSO by performing the following steps:

  Procedure

  1. After the stage-1 configuration is deployed, log in to the device.
  2. Log in to the Junos OS CLI and access configuration mode.
  3. Update the source NAT rule configuration on the device to add interface NAT. The following is a sample configuration.
     set security nat source rule-set deop1-1-lan-wan-ruleset from routing-instance trust

     set security nat source rule-set deop1-1-lan-wan-ruleset to interface ge-1/0/1.0

     set security nat source rule-set deop1-1-lan-wan-ruleset to interface ge-1/0/2.0

     set security nat source rule-set deop1-1-lan-wan-ruleset to interface ge-1/0/9.0

     set security nat source rule-set deop1-1-lan-wan-ruleset rule rule-WAN_0 match source-address 0.0.0.0/0

     set security nat source rule-set deop1-1-lan-wan-ruleset rule rule-WAN_0 then source-nat interface

     set security nat source rule-set deop1-1-lan-wan-ruleset rule rule-WAN_1 match source-address 0.0.0.0/0

     set security nat source rule-set deop1-1-lan-wan-ruleset rule rule-WAN_1 then source-nat interface

  Bug Tracking Number: CXU-24606.

• The Help (?) menu for the Administration Portal and Customer Portal is empty and does not display any links.

  Workaround: Use the following links to access the Administration Portal and Customer Portal Help Centers:

  • Administration Portal: https://www.juniper.net/documentation/en_US/cso4.0/help
    /information-products/pathway-pages/admin-portal-index.html

  • Customer Portal: https://www.juniper.net/documentation/en_US/cso4.0/help/information-products
    /pathway-pages/customer-portal/cp-index.html

  You can also use the More... hyperlink on a page to access the online help content for that page.

  Bug Tracking Number: CXU-24941.

• For an NFX250 device, the Ubuntu VNF service chain configuration is incorrect if you set SINGLE_SSH_TO_NFX to False and then instantiate a service.

  Workaround: None.

  Bug Tracking Number: CXU-25018.

• In CSO Release 4.0.1, while you are converting a hub that is created in Release 3.3.1 to a secure OAM hub by using the stage-2 template, the job fails even though the device configuration is updated.

  Workaround: Roll back the stage-2 configuration, save the stage-2 configuration, and redeploy.

  Bug Tracking Number: CXU-25531.

• After you upgrade CSO from Release 3.3.1 to Release 4.0.1, the widgets are not displayed in the Dashboard.

  Workaround: You must clear the preferences in Redis.

  To clear the preferences in Redis:

  1. Log in to the central microservice VM.

  2. Execute the redis-cli command.

  3. Execute the FLUSHALL command.

  Bug Tracking Number: CXU-25405.

• The upgrade from CSO Release 3.3.1 to Release 4.0.1 fails, because the pods do not get deleted.

  Workaround: Delete the pods and rerun the upgrade.sh script.

  1. Log in to the microservice VM.

  2. Execute the kubectl delete deployments,svc,pods,ds,events --all --grace-period=0 --force command.

  3. After the command is successfully executed, rerun the upgrade.sh script.

  Bug Tracking Number: CXU-25737.

• While upgrading CSO from Release 3.3.1 to Release 4.0.1, if link-interface or link-name mappings in Redis are lost, the link utilization graph is not displayed.

  Workaround: None.

  Bug Tracking Number: CXU-25728.

• When you generate reports for operating companies and their tenants, the customized logo is not displayed.

  Workaround: None

  Bug Tracking Number: CXU-24729.

• In CSO Release 4.0.1, the upgrade fails for sites with vSRX as a CPE device.

  Workaround: Before upgrading the site, manually execute the request system storage cleanup command on the vSRX device CLI.

  Bug Tracking Number: CXU-24729.

• The operation to delete tenants fails and the following error is displayed:

  PMTask Fail {"status_code": "409", "error_tag": "Refs Exist", "error_message": "Operation can not be completed since the resource is still being referred to. ", "error_app_message": "Delete when resource still referred: [u'/topology-service/termination-point/edea81b3-fc95-4d19-beea-e56ecfce3fb3', u'/topology-service/termination-point/1ba769cc-00d6-4ad3-914e-913cb4d707eb', u'/topology-service/termination-point/a74ff397-6993-437f-ba67-4ef0d57620bc', u'/topology-service/termination-point/b9a5dd21-3025-4cce-b8a9-7df6d32cf51f', u'/topology-service/termination-point/89c9f27d-4f68-4972-9c10-7ab68805d146', u'/topology-service/termination-point/6afba242-6905-4d3e-8295-4b7b5e791fdd', u'/topology-service/termination-point/e627802b-ec97-4343-8f5c-4c53415d3353']", "error_diag": "This error occurs when a resource is attempted for modification or deletion when there are child resources still referring to it. ", "error_code": "40006"}

  Workaround: Delete the termination points by using the REST API and then retry the delete operation.

  Bug Tracking Number: CXU-25686.

• The ZTP of an NFX250 device fails.

  Workaround: Do not configure hub sites in parallel. Create a new hub site only after you have completed provisioning the previously created hub site.

  Bug Tracking Number: CXU-25822.

• In device redundancy mode, while creating a NAT pool, the Routing Instance field is not displayed.

  Workaround: None.

  Bug Tracking Number: CXU-25880.

• The activation of an NFX250 as an SD-WAN CPE device with an ADSL or VDSL WAN link fails if you set the global parameter USE_SINGLE_SSH of the device template to False.

  Work around: In the device template, set the global parameter USE_SINGLE_SSH to True and onboard the site.

  Bug Tracking Number: CXU-25573.

• On the Monitor Overview Page page, a green check mark appears even though the operational state of a device that is going through the RMA process is Down, Expected, or RMA.

  Workaround: None.

  Bug Tracking Number: CXU-24933.

• On the Audit Logs page, Username and Role columns do not display the actual name and the role of the user, respectively. Instead, the name of the user is displayed as Admin and role of the user is displayed as _member_.admin.

  Workaround: None.

  Bug Tracking Number: CXU-25189.

• An error occurs while EEPROM contents for copper ports are being read.

  Workaround: None.

  Bug Tracking Number: PR1372217.

• Even though you have successfully upgraded the software image for an NFX250 device, the Jobs page displays the status as Failed.

  Workaround: None.

  Bug Tracking Number: CXU-25860.

• On an MX Series device, after you apply the stage-2 configuration, the BGP route is hidden.

  Workaround: Create a static route in the provider edge (PE) router to reach the loopback IP address of the spoke device.

  Bug Tracking Number: CXU-25665.

• For a device that is provisioned for an OpCo tenant, the software image upgrade fails if you try to upgrade the software image from the Device Images page.

  Workaround: None

  Bug Tracking Number: CXU-25663.

• The POP location on the geographical map is incorrect if you do not validate the address while you are creating a POP.

  Workaround: After you specify the address on the Add POP page, click Validate before you proceed to the next page.

  Bug Tracking Number: CXU-25060.

• While you are editing a role, when you clear an object, the implied capabilities that are listed under the object are cleared.

  Workaround: Do one of the following:

  • Before you submit the changes, identify the implied capabilities. Capabilities with an icon (i) are implied capabilities.

  • After you submit the changes, preview the role to verify implied capabilities.

  Bug Tracking Number: CXU-25386.

• Because of insufficient buffer size, vSRX performs queue scheduling incorrectly and drop packets.

  Workaround: Set the buffer size to 3000 microseconds by executing the set class-of-service schedulers scheduler-name buffer-size temporal 3000 command.

  Bug Tracking Number: PR1361720.