Known Issues

Device Life-Cycle Management

• If the device onboarding fails, the device onboarding status is displayed as Status is not available in the Devices section of the Network Implementation Plan page (Inventory > Device Onboarding > Network Implementation Plan).

  Workaround: For such devices, initiate the outbound SSH connection on the router to restart the onboarding workflow.

• In this release, single node failure and restart might cause inconsistencies in device inventory.

  Workaround: None.

• Paragon Automation triggers the configuration templates included in a device profile and interface profile only during the initial onboarding of the device. You cannot use the configuration templates included in the device profiles and interface profiles to apply additional configuration on a device after the device is onboarded.

  Workaround: If you need to apply additional configuration on a device after the device is onboarded, you need to manually apply the configuration using the CLI.

• If you have enabled the Trust option in the device profile, the onboarding workflow may fail with the following error:

  Onboarding workflow failed reason: task_failure, Trust score computation failed.

  Workaround: Retry the onboarding process after a few minutes.

• Onboarding an ACX7024 device fails with the following error:

  task_failure, Failed to launch Software update. Timed out or waiting for the launch message. Try again later.

  Workaround: Restart the onboarding process using the service orchestration cMGD CLI. Perform the following steps:

  1. Log into the primary node using SSH.

  2. Exit out of the default paragon CLI to the Linux root shell.

  3. Log in to the service orchestration CMGD CLI.

  4. Get the organization UUID from the GUI. To find your organization's UUID, go to System Settings > Organization Settings on the Paragon Automation UI banner and then copy the organization's UUID.

  5. Configure the organization UUID in the cMGD CLI:

  6. Retrieve the service order.

  7. Reset the node state.

  8. Log in to the router.

  9. Restart the onboarding by restarting the connection.

• The onboarding of a device fails if you configure only IPv6 addresses at the time of installation and enable Trust in the device profile. This issue does not occur if you have configured IPv4 addresses.

  Workaround: To successfully onboard the device, do one of the following:

  • Disable Trust in the device profile and retry onboarding. In this case, the Trust compliance score will not be calculated.
  • At the time of installation, include IPv4 addresses in addition to IPv6 addresses.

• The View Network Resources page (Inventory > Device Onboarding > Network Implementation Plan > More) doesn't display AE interfaces-related details.

  Workaround: You can view the AE interfaces-related details in the View active config link of the Configuration accordion (Observability > Troubleshoot Devices > Device-Name).

Observability

• The Hardware accordion does not display the following information for the listed devices:

  • Chassis temperature (chassis-temperature) for MX204, MX240, MX304, MX10004, MX10008, and MX10016 devices.

  • Charts related to the fan speed (rpm-percent) for MX480, MX960, MX10004, MX10008, and MX10016 devices.

  • Power supply module temperature (psm-temperature) for MX204, MX480, and MX960 devices.

  • CPU utilization (cpu-utilization) information is not available for PTX10002 device.

  • Line card charts are not available for some ACX Series, MX Series, and PTX Series devices as the flexible PIC concentrator (FPC) fields are not supported on these devices. See Table 1 for more information.

    Table 1: Line Card Charts Support

    Device Family	Device Series	FPC Fields Not Supported
    ACX Series	ACX7100-32C, ACX7100-48L, ACX7024, ACX7024X, ACX7509, ACX7348	fpc-temperature, fpc-cpu-utilization, fpc-buffer-memory-utilization
    MX Series	MX204, MX240, MX304, MX480, MX960, MX10004, MX10008, MX10016	fpc-temperature, fpc-cpu-utilization
    PTX Series	PTX10002	fpc-temperature, fpc-cpu-utilization, fpc-buffer-memory-utilization

• On the Interfaces accordion, forward error correction (FEC) corrected errors and FEC uncorrected errors charts are available only on interfaces that support speeds equal to or greater than 100-Gbps.

• After you apply a new configuration for a device, the Active Configuration for Device-Name page (Observability> Troubleshoot Device > Device-Name > Configuration accordion > View active config link) does not display the latest configuration immediately. It takes several minutes for the latest changes to be reflected on the Active Configuration for Device-Name page.

  Workaround: You can verify whether the new configurations are applied to the device by logging in to the device using CLI.

• The graphs related to cyclic redundancy check (CRC) errors display No Results Found as the CRC errors-related data is not streamed from the devices for the management ports.

  Workaround: None.

• If a device is discovered through a BGP-LS peering session even before you onboard the device, then duplicate LSPs are created when a PCEP session is established with the device. In some rare cases, the duplicate LSPs may remain.

  Workaround: If you see duplicate LSPs, restart the EdgeAdapter pod.

• After the primary node is switched off, the OC-term and GNMI state on the Remote Management accordion (Observability > Troubleshoot Devices > Device-Name) are displayed as disconnected.

  Workaround: You can do one of the following:

  • Offboard and onboard the devices, or

  • Restart the OC-term pod.

• Sometimes, the graph on the Output Traffic Details for Device-Name page (Observability > Health >Troubleshoot Devices > Device-Name > Overview > Interfaces (accordion) > Output Traffic data link) displays a sudden spike in data. This issue occurs because some devices might erroneously send data more than once with the same time stamp.

  Workaround: None.

• The number of unhealthy devices listed on the Troubleshoot Devices and Health Dashboard pages (Observability > Health) do not match.

  Workaround: None.

• You cannot delete unwanted nodes and links from the Paragon Automation GUI.

  Workaround: Use the following REST APIs to delete nodes and links:

  • REST API to delete a link:

    [DELETE] https://{{server_ip}}/topology/api/v1/orgs/{{org_id}}/{{topo_id}}/links/{{link_id}}

    Note:

    You can follow the steps described here to get the actual URL.

    For example,

    • URL: 'https://10.56.3.16/topology/api/v1/orgs/f9e9235b-37f1-43e7-9153-e88350ed1e15/10/links/15'

    • Curl:

  • REST API to delete a node:

    [DELETE] https:// {{Server_IP}}/topology/api/v1/orgs/{{Org_ID}}/{{Topo_ID}}/nodes/{{Node_ID}}

    Note:

    You can follow the steps described here to get the actual URL.

    For examples,

    • URL: ' https://10.56.3.16/topology/api/v1/orgs/f9e9235b-37f1-43e7-9153-e88350ed1e15/10/nodes/1'

    • Curl:

    Use the following procedure to get the actual URL that you use in CURL for deleting a link or a node:

    1. Navigate to the Topology page (Observability > Topology).

    2. Open the developer tool in the browser by using the CTRL + Shift + I buttons in the keyboard.

    3. In the developers tool, select Network and select the XHR filter option.

    4. Identify the link index number or node number. To identify the link index number to the node number:

       1. On the Topology page of the Paragon Automation GUI, double click the link or the node that you want to delete.

          The Link Link-Name page or the Node Node-Name page appears.

       2. Navigate to the Details tab and note the link index number or the node number that is displayed.

    5. In the developers tool, select and click the row based on the link index number or the node number that is related to the link or the node that you want to delete.

    6. Copy the URL that you need to use to delete the link or node in CURL.

  • The values for Input Utilization and Output Utilization are represented in terms of percentage of interface utilization instead of Mbps.

    This issue is seen in Physical Interfaces Details for Service-Instance-Name tab (Orchestration > Instances > Service Instances > Service-Instance-Name hyperlink > Service-Instance-Name Details > Passive Assurance ) and Input Traffic Details for Device-Name page (Observability > Health > Troubleshoot Devices > Device-Name > Overview > Interfaces (accordion) > Input Traffic data-link).

    Workaround: To get the actual traffic rate value, you should multiply the values displayed by interface speed.

• Not all optics modules support all the optics-related KPIs. See Table 2 for more information.

  Workaround: None.

  Table 2: KPIs Supported for Optics Modules

  Module	Rx Loss of Signal KPI	Tx Loss of Signal KPI	Laser Disabled KPI
  SFP optics	No	No	No
  CFP optics	Yes	No	No
  CFP_LH_ACO optics	Yes	No	No
  QSFP optics	Yes	Yes	Yes
  CXP optics	Yes	Yes	No
  XFP optics	No	No	No

• For PTX100002 devices, the following issues are observed on the Interface accordion (Observability > Health > Troubleshoot Devices > Device-Name > Overview):

  • On the Pluggables Details for Device-Name page (Interfaces accordion > Pluggables data-link), the Optical Tx Power and Optical Rx Power graphs do not display any data.

  • On the Input Traffic Details for Device-Name page (Interfaces accordion > Input Traffic data-link), the Signal Functionality graph does not display any data.

  • On the Interfaces Details for Device-Name Page (Interfaces accordion > Interfaces data-link), the link flaps count erroneously drops to zero and reverts to the same count as before.

    Workaround: None.

• For EX9204 devices, the Optical Tx power and Optical Rx Power graphs on the Pluggables Details for Device-Name page (Observability > Health > Troubleshoot Devices > Device-Name > Overview> Interfaces accordion > Pluggables data-link) do not show any warnings for critical thresholds.

  Workaround: None.

• For EX4300-48MP devices, the Signal Functionality graph on the Input Traffic Details for Device-Name page (Observability > Health > Troubleshoot Devices > Device-Name > Overview> Interfaces accordion > Pluggables), does not display any data.

  Workaround: For Pluggables (Optical Interfaces), we recommend that you install Junos OS Release 24.4R2 on your device.

• While creating or editing an LSP, the Routing Path field (Add or Edit Tunnel > Path tab) for a node may not list IPv6 addresses. However, you should be able to select any Link or Node to create the path.

  Workaround: None.

• You won’t be able to provision an LSP using the Paragon Automation GUI if the provisioning type is SRv6.

  Workaround: We recommend that you provision SRv6 LSPs using the REST APIs.

• The changes are not reflected when you update the following fields on the Constraints tab of the Edit LSP page:

  • Admin Group Include All

  • Admin Group Include Any

  • Admin Group Exclude

  This issue occurs only when you try to edit an LSP that doesn’t have at least one of these attributes set at the time of LSP creation.

  Workaround: None.

• If you have installed a Junos OS or Junos OS Evolved release other than Release 22.1R1.10, you may not be able to view the historical data for link delays. For any other versions of Junos OS or Junos EVO release, you may see discrepancies in the graph as delay statistics are sent only when there are changes.

  Workaround: None.

• Your hard disk may be full because of the LSP event-related data retention.

  Workaround: If you have enabled the PCEP protocol, we recommend that you monitor the disk usage and clean up the unused LSP events-related data in the Postgres database. To do the cleanup:

  1. Check the disk usage utilization of the Postgres database.

     1. SSH to one of the primary nodes.

     2. Type exit to exit from Paragon Shell.

     3. Run the paragon-local-volume-check command.

     4. Look for the size of pgdata-atom-db* and see if it has grown over time.

  2. Clean up the unused LSP event-related data, in case of highly-utilized disk due to Postgres usage.

     1. SSH to one of the primary nodes

     2. Type exit to exit from Paragon Shell.

     3. Run the paragon-db command.

     4. Run the \c ns_pcs; command.

     5. Run the truncate table pcs_lsp_event; command.

     If a read-only error occurs, you can choose a different primary node, depending on the role you have for connecting to Postgres,

• The historical data for tunnel traffic (Observability > Topology > Tunnels > View >Tunnel Traffic) is not displayed for ACX Series devices.

  Workaround: None.

Service Orchestration

• The "vpn_svc_type" service type is displayed as "pbb-evpn" instead of "evpn-mpls" on the Paragon Automation GUI and through the REST API.

  Workaround: None.

• Sometimes, the apply insights configuration (appy_insights-config) fails if you try to provision a service without properly deleting a previously provisioned service or a device.

  For example, if you release the router without off-boarding or deleting a service, then the apply insights configuration fails when the same service or device is used in another organization.

  Workaround:

  • If there are stale services and devices, run the following REST APIs from the cMGD container of the foghorn namespace to delete stale services and devices, and rerun the workflow:

    • curl --request DELETE <http://config-server.healthbot:9000/api/v2/config/services/device-group/<device-group> name>/

    • curl --request DELETE <http://config-server.healthbot:9000/api/v2/config/device-group/<device-group> name>/

    • curl --request POST <http://config-server.healthbot:9000/api/v2/config/configuration>/

  • If there are stale network-groups, run the following REST APIs from the cMGD container of the foghorn namespace to delete the stale network-groups, and rerun the workflow:

    • curl --request DELETE <http://config-server.healthbot:9000/api/v2/config/services/device-group/<network-group> name>/

    • curl --request DELETE <http://config-server.healthbot:9000/api/v2/config/device-group/<network-group> name>/

    • curl --request POST <http://config-server.healthbot:9000/api/v2/config/configuration>/

• The EVPN service order creation fails if you try to create an EVPN service order by importing an existing JavaScript Object Notation (JSON) file.

  Workaround: If you are using a JSON file, ensure that you clear the placement section before you publish the service order.

• For some devices such as ACX7204, if you configure VLANs on unused ports, the following error occurs:

  VLAN must be specified on tagged interfaces.

  Workaround: This issue is caused by the default factory configuration on the port. Delete the default factory configuration on the ports that you plan to use.

• For an MX 240 device, the OSPF-related data is not populated on the Passive Assurance tab (Orchestration > Instances > Service-Order-Name Details).

  Workaround: Configure OSPF on the customer edge (CE) device.

• Although multiple VLAN IDs are available in the topology resources, the Placement section of the EVPN service order lists only one VLAN ID in the drop down.

  Workaround: To fix this issue:

  1. Edit the EVPN service order to add new VLAN IDs. You can add the VLAN IDs under the Tagged Interface section.

  2. Clear the Placement section by deselecting the device name.

  3. Save and publish the service order.

• While modifying a service order, you cannot clear the existing placements.

  Workaround: If publishing the service order fails due to existing placements, you can export the failed service order to JSON format and then create a new service order or modify an existing service order by importing this JSON file. During the importing process, delete the placements and then publish the service order.

• While creating or modifying an EVPN service order, you cannot configure multiple VLAN IDs on the Aggregated Ethernet (AE) interface. The EVPN considers the AE port as a single resource and therefore an AE interface cannot be reused across service instances even when the VLAN IDs on the AE IFL differ.

  Workaround: None.

• VLAN drop down under Placement section doesn't display all the available VLANs as per the topology service order and it shows only the selected VLAN.

  Workaround: None.

• During device onboarding, pings from the device to Microsoft Azure and Google Cloud Platform endpoints fail.

  Workaround: None.

• While creating an EVPN service order, if the MAC Address Limit that you have specified is out of the defined range then the service order fails.

  Workaround: Specify a value that is within the defined range and then republish the service order.

• While creating or modifying an EVPN service order, the MAC Address Limit configuration is ignored if you specify Drop as the action to be taken when the upper limit for customer MAC addresses exceed.

  Workaround: None.

• Since you can manage resources from within network implementation plans, a blank topology instance (topo) is auto created on the Resources Instances page (Orchestration > Service > Resource Instances). This topology instance is a read-only instance and is owned by the network operator.

  Workaround: None.

• If you have referenced a device in a service order and if you try to release the router from the organization, then you cannot delete the service.

  Workaround: We recommend that you deprovision the service and offboard the device before you release the router from the Network Inventory page.

• When you click the Refresh icon on the Service-Instance-Name Details page (Orchestration > Instances > Service-Instance-Name), you may not see the latest events in the Relevant Events section.

  Workaround: To view the latest events, instead of using the Refresh icon go to the Service Instance page (Orchestration > Instances) and select the service instance for which you need to see the latest events.

• After you upgrade Paragon Automation from Release 2.1.0 to Release 2.2.0, L2VPN and L3VPN accordions (Orchestration > Instances > Service Instances > Service-Instance-Name hyperlink > Service-Instance-Name Details > Passive Assurance tab) show no data for service instances that were provisioned in Release 2.1.0. This issue does not occur if you install Paragon Automation afresh.

  Workaround: You need to recreate the VPN service after you upgrade to Juniper Paragon Automation Release 2.2.0.

• While modifying an existing L3VPN service instance, if you try to remove a device that is already a part of a network implementation plan then the modify workflow fails.

  Workaround: On the Monitors page, stop all the monitors associated with the device that has to be deleted in the service instance. After the relevant monitors are stopped, you can proceed with modifying the L3VPN service instance.

• Scheduling provisioning of service orders does not work if you upgrade Paragon Automation to Release 2.2.0.

  Workaround: None.

• The Order History tab on the L3VPN-Name Details page (Orchestration > Instances > Service-Instance-Name hyperlink) lists all the order history even if the service instance is deprovisioned and provisioned again.

  Workaround: None.

• When you upgrade an L2 circuit service design, you are prompted to update the L2 address even when there is no dependency on L2 addresses.

  Workaround: Before you upgrade an L2 circuit service design that is associated with a service instance, you must upgrade the L2 address, VPN resources, and topology services using the Paragon Automation GUI.

• While creating topo resources, the node name is not populated. This issue occurs only after you upgrade to Release 2.3.0.

  Workaround: Perform the following steps to ensure that the node name is populated:

  1. Navigate to the Troubleshoot Devices page (Observability > Health) or Network Inventory (Inventory > Devices) page.

  2. Select one or more devices for which the node name is not populated and click More > Assign to Site.

  3. On the Assign Devices to Sites page, select a site other than the site the device was originally assigned to.

  4. After you assign a new site, repeat Step 1 through Step 3 and assign the device to the original site.

• In a scaled setup, you cannot upgrade service designs in bulk.

  Workaround: We recommend that you upgrade only one service design at a time.

• The Output Traffic rate column on the Logical Interface accordion (Orchestration > Instances > Service Instances page > service-instance-name hyperlink > Service-Instance-Name Details) displays some data even when there is no traffic through the devices.

  Workaround: None.

• After you upgrade Paragon Automation to Release 2.3.0 and upgrade the L3VPN service instance to the latest service design, you may not see the physical interface of one of the devices.

  Workaround: Restart the AED pods by running the following command:

  kubectl delete pod -n healthbot analytical-engine-pod-names

  Ensure that the pod names are separated by a space.

  To get the list of analytical-engine pods names, run the following command:

  kubectl get pods -n healthbot | grep analytical-engine

• You cannot provision an L3VPN service on an ACX7024 device with Junos OS Release 23.2R2 installed.

  Workaround: None.

Active Assurance

• The status of a Test Agent is shown as offline after the device's Routing Engine switches over from the primary Routing Engine to the backup Routing Engine, or vice versa. This issue occurs only if you are using a Junos OS version that is older than 23.4R2.

  Workaround: Reinstall Test Agent after the Routing Engine switchover.

• You cannot run multiple versions of plug-ins on a Test Agent.

  Workaround: When you upgrade Paragon Automation, restart all measurements before creating any new measurements.

• When you click a Monitor on the Monitors page (Observability > Active Assurance), the Monitor-Name page takes approximately a minute to load the data. This issue occurs only when there are more number of events in the system.

  Workaround: None.

• The streams are not generated when you create a Test with a DNS plug-in and the following event is raised:

  Could not get nameserver from resolv.conf

  This issue occurs when the Test is associated with a Test Agent that runs on a Juniper Networks router with Junos OS EVO installed, and you don't specify the Name Server field while configuring a Test.

  Workaround: Ensure that you specify a value for the Name Server field while configuring a Test.

• After you update a Monitor or a Test Template that is created by another user, the Updated By column on the Monitors (Observability > Active Assurance) and Test Template (Inventory > Active Assurance) pages do not reflect the name of the user that modified the Monitor or the Test Template.

  Workaround: None.

• When you add a new host to the existing Monitor, the new measurements are not reflected in the Active Assurance tab of the Health Dashboard (Observability > Health).

  Workaround: None.

• The devices table on the Devices tab (Observability > Health > Health Dashboard > Active Assurance (Tab) > Click any accordion > View Details > Affected Items tab) does not list devices that have unhealthy measurements.

  Workaround: None.

Trust

There are no known issues in this release.

Administration

There are no known issues in this release.

Installation and Upgrade

• The backup and restore functionality has the following caveats:

  • You cannot restore data backed up from a setup with a different release of Paragon Automation to a setup with the current release.

  • Paragon Automation backs up only application configurations such as devices, sites, service orders, and so on. Since a backup does not store the certificates and infrastructure services configurations, that information must be kept unchanged during restoration.

  • Resources allocated to the network won’t be preserved after a restore and you must ensure that you release the allocated resources during the window between taking a backup and performing a restore.

  Workaround: None.

• If a node in the cluster is not operational, the status of the vector pod from the node that is not operational is displayed as Running, even though the node status is reported as Not Ready. This is due to an existing Kubernetes issue. See https://github.com/kubernetes/kubernetes/issues/117769.

  Workaround: You can do one of the following:

  • Monitor the metric, kube_daemonset_status_number_ready. When the value for this metric drops to 3, you can manually check from which vector the data is missing.

  • Set a query and an alert for the kube_daemonset_status_number_ready metric in Grafana.

• You might encounter RKE2-related issues if you change the hostname after you set up a cluster.

  We recommend that you do not change the hostname after a cluster is set up.

  Workaround: None.

• When the worker node is down, there might be issues if you create an organization or onboard a device.

  Workaround: Do not create an organization or onboard a device when a worker node is down. You must wait until the cluster recovers and then create an organization or onboard a device. Recovered state is when all the pods are either in Running or Pending states and are not in any intermediate states like Terminating, CrashloopbackOff, and so on.

• If you have powered off one of the primary nodes, you may not be able to log in to the Juniper Paragon Automation GUI.

  Workaround: Restart papi-ws using the following Paragon Shell CLI command:

  request paragon cluster pods reset service papi-ws namespace papi operation restart

• Upgrading from Release 2.1.0 to Release 2.2.0 on VMs configured with minimum required specifications might fail due to a failed airflow-worker instance. Verify the reason for failure.

  Workaround: Perform the following steps:

  1. Lower the following CPU resource requests from the Linux root shell.

     # kubectl patch deployment -n mems mems --type "json" -p '[{"op":"add","path":"/spec/template/spec/containers/0/resources/requests/cpu","value":"0.5m"}]'# kubectl patch deployment -n tagify tagify --type "json" -p '[{"op":"add","path":"/spec/template/spec/containers/0/resources/requests/cpu","value":"0.5m"}]'

  2. Rerun upgrade.