Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

Sensors Overview

 

Paragon Insights accepts data from a variety of Juniper and third-party devices and from various types of telemetry sensors that include traditional network management protocols like syslog and SNMP. Paragon Insights supports push and pull models of data collection. In the push model, your devices push telemetry data to Paragon Insights through trap notifications, for instance. In the pull mode, Paragon Insights periodically polls your devices for data. This guide describes each of the supported ingest methods, with examples, sorted by whether they fall into the push or pull model. Along with each description, we provide the required Junos OS version and device configurations needed to enable the specific ingest type.

As the number of objects in the network, and the metrics they generate, have grown, gathering operational statistics for monitoring the health of a network has become an ever-increasing challenge. Traditional ’pull’ data-gathering models, like SNMP and the CLI, require additional processing to periodically poll the network element, and can directly limit scaling.

The ’push’ model overcomes these limits by delivering data asynchronously, which eliminates polling. With this model, the Paragon Insights server can make a single request to a network device to stream periodic updates. As a result, the ’push’ model is highly scalable and can support the monitoring of thousands of objects in a network. Junos devices support this model in the form of the Junos Telemetry Interface (JTI).

Paragon Insights currently supports five push-model sensors:

While the ’push’ model is the preferred approach for its efficiency and scalability, there are still cases where the ’pull’ data collection model is appropriate. Two examples might be when a device doesn’t support the Junos Telemetry Interface (JTI), or when managing third party devices. With the pull model, Paragon Insights requests data from network devices at periodic, user-defined intervals.

Paragon Insights currently supports the following pull-model sensors:

Native GPB

Native sensors use a Juniper-proprietary data model using Google protocol buffers (GPB). The device pushes telemetry data (when configured) over UDP.

The device pushes data from the Packet Forwarding Engine, that is, directly from a line card or a routing engine. This means telemetry data is sent over the forwarding plane, so the collector must have in-band connectivity to the device.

To use native format, you configure the device with settings that include where to send the telemetry data. When you configure Paragon Insights to start collecting the data, the stream is already flowing towards the server. Native sensors can send telemetry data close to the source such as physical interface statistics or statistics for label-switched paths (LSPs). A sensor starts to emit data as soon as it is enabled.

For more information on native sensors, see Understanding the Junos Telemetry Interface Export Format of Collected Data.

To know how to configure a Junos OS device to use Native GPB, see Configure Native GPB Ingest.

NetFlow

Paragon Insights supports NetFlow v9 and NetFlow v10 (IPFIX) natively as NetFlow ingest method, using a data model that aligns with other Paragon Insights ingest mechanisms. NetFlow is a network protocol for collecting IP traffic statistics, which can then be exported to a tool for analysis. The NetFlow v9 data export format is described in RFC 3954; NetFlow v10 is officially known as IPFIX and standardized in RFC 7011.

Junos devices support flow monitoring and aggregation using these protocols; the Junos OS samples the traffic, builds a flow table, and sends the details of the flow table over a configured UDP port to a collector, in this case Paragon Insights. Paragon Insights receives the incoming Netflow data, auto-detects it as v9 or v10, and process it further.

As shown above, the network device pushes data from the Packet Forwarding Engine, that is, directly from a line card. This means flow data is sent over the forwarding plane, so the collector must have in-band connectivity to the device. To use the flow sensor option, you configure the device with settings that include where to send the flow data. When you configure Paragon Insights to start collecting the data, the flow data is already flowing towards the server.

Paragon Insights uses flow templates as a mechanism to identify and decode incoming flow data before sending it for further processing. Paragon Insights provides predefined flow templates for NetFlow v9 and v10 (IPFIX), or you can define your own. The predefined templates match those which the Junos OS currently supports. For example, the Junos OS template, ipv4-template, aligns with the Paragon Insights template hb-ipfix-ipv4-template. To view the fields used in the Junos OS templates, see Understanding Inline Active Flow Monitoring.

Note

In the current ingest implementation for NetFlow, the following field types are not supported:

  • Fields for enterprise specific elements

  • Variable length fields

Warning

For NetFlow ingest, ensure that there is no source NAT in the network path(s) between the device and Paragon Insights. If the network path contains source NAT, then the received device information is not accurate.

A typical workflow includes adding a NetFlow configured device in Paragon Insights, configuring NetFlow templates, configuring a rule with Flow sensor, and deploying a playbook with the rule to a device group.

With the playbook applied, you can begin to monitor the devices.

  1. Click Monitoring > Network Health in the left navigation bar and click on the Device Group tab.
  2. Select the device group to which you applied the playbook from the Device Group pull-down menu.
  3. Select one of more of the devices to monitor.
  4. In the Tile View, the external tile contains the parameters from the rule you configured earlier.

sFlow

Paragon Insights supports sFlow (v5) natively as another flow-based ingest method.

sFlow is a statistical sampling-based technology for high-speed switched or routed networks. You can configure sFlow to continuously monitor traffic at wire speed on all interfaces simultaneously if you want.

sFlow provides or helps with:

  • Detailed and quantitative traffic measurements at gigabit speeds

  • Insight into forwarding decisions

  • Troubleshooting for network issues

  • Congestion control

  • Security and audit trail analysis

  • Route profiling

Everything that sFlow does above, it does without impact to forwarding or network performance. For more information on sFlow, see: RFC 3176, InMon Corporation's sFlow: A Method for Monitoring Traffic in Switched and Routed Networks.

As a statistical sampling protocol, Juniper’s sFlow agent samples the traffic and counters on network interfaces, creates sFlow datagrams and forwards them to external sFlow collectors. Paragon Insights is one such collector.

To know how to configure sFlow packets in Paragon Insights, go to Configure sFlow Settings.

OpenConfig

To use OpenConfig format, you configure the device as a gRPC server. With Paragon Insights acting as the client, you define which sensors you want it to subscribe to, and it makes subscription requests towards the device.

Data streamed through gRPC is formatted in OpenConfig key/value pairs in protocol buffer (GPB) encoded messages. Keys are strings that correspond to the path of the system resources in the OpenConfig schema for the device being monitored; values correspond to integers or strings that identify the operational state of the system resource, such as interface counters. OpenConfig RPC messages can be transferred in bulk, such as providing multiple interface counters in one message, thereby increasing efficiency of the message transfer.

For more information on OpenConfig sensors, see Understanding OpenConfig and gRPC on Junos Telemetry Interface.

gNMI-Encoded OpenConfig RPC

gNMI-encoded OpenConfig works much like OpenConfig RPC in that you must set the network device up as an OpenConfig server to which Paragon Insights makes subscription requests. However, gNMI supports more subscription types than Paragon Insights currently supports. Currently, Paragon Insights only supports gNMI STREAM subscriptions in the SAMPLE mode. STREAM subscriptions are long-lived subscriptions that continue, indefinitely, to transmit updates relating to the set of paths configured within the subscription. SAMPLE mode STREAM subscriptions must include a sample_interval.

The messages returned to the client through gNMI are encoded by the device in protobuf, JSON, or JSON-IETF format and cannot be sent in bulk. This, in part, makes gNMI-encoded messaging less efficient than gRPC-encoded messaging.

Warning
  • For JSON or JSON-IETF, it is assumed that the device returns gNMI updates as only leaf values encoded in JSON, rather than returning an entire hierarchy or sub-hierarchy as a JSON object.

  • Numbers encoded in JSON or JSON-IETF are decoded by Paragon Insights as either float64, int64, or string, according to RFC 7159 and RFC 7951. If your OpenConfig rules contain fields that are of a different type, we recommend that you change the field types accordingly.

Junos OS and Cisco devices can be managed by Paragon Insights using gNMI-encoded OpenConfig. If a device does not support gNMI in general, or the STREAM subscription in SAMPLE mode, or does not support an OpenConfig request, it returns one of the following errors:

  • Unimplemented

  • Unavailable

  • InvalidArgument

In the case of a Junos OS or Cisco device, the error causes the connection to fall back to OpenConfig RPC. In the case of a third-party device, the connection simply fails due to the error.

gNMI-encoded OpenConfig can be enabled at the device or device-group level. If enabled at the device-group level, then all devices added to the group use gNMI by default. If enabled (or not enabled) at the device level, then the device level setting takes precedence over the device-group level setting.

Note

During the initial connection gNMI devices attempt to perform an initial sync with the client. The device sends a continuous stream of data until the device and the collector (Paragon Insights) are in sync. After initial sync, the device begins normal streaming operations based on the configured reporting rate. Because of the processing load this creates, Paragon Insights has this feature disabled by default. It can be enabled at the device-group or device level if needed.

For more information about gNMI, see: gRPC Network Management Interface (gNMI).

Device Configuration for OpenConfig

OpenConfig requires:

  • Junos OS Version: 16.1 or later

    • The OpenConfig sensor requires that the Junos device have the OpenConfig and network agent packages installed. These packages are built into Junos OS Releases 18.2X75, 18.3, and later. For releases between 16.1 and 18.2X75 or 18.2, you must install the packages.

      To verify whether you have these packages, enter the following command:

      user@host> show version | match "Junos:|openconfig|na telemetry"

      See Understanding OpenConfig and gRPC on Junos Telemetry Interface for more information.

    • Network agent is not supported on PPC platforms (MX104, MX80, and so on).

Device Configuration

Configure a device by entering the following command:

set system services extension-service request-response grpc clear-text port <port number>

To configure OpenConfig port under device configuration in Paragon Automation GUI, see Editable Fields on the Edit Devices Page table in Edit Devices.

Syslog

In addition to the JTI-related options above, Paragon Insights also supports system logs as a data collection method, using a data model that aligns with other ingest mechanisms.

A device can push syslog messages (when configured) over UDP to the Paragon Insights server either out-of-band through the Routing Engine (RE) using the router’s management interface, or in-band through the Packet Forwarding Engine, that is, directly from a line card.

To use syslog format, you configure the device with settings that include where to send the syslog messages. When you configure Paragon Insights to start collecting the data, messages are already flowing towards the server.

For more information on syslog as used by Juniper devices, see Junos OS System Log Overview.

A syslog message consists of a header, structured data in key-value format within square brackets, and the log message. The header consists of the following information:

  • Log priority

  • Version number of Syslog protocol specification on header format.

    Currently, this number is 1.

  • Timestamp of when the message was generated in the format of "Mmm dd hh:mm:ss".

  • Hostname identifies the device that sent the syslog message.

  • Application name

  • Application process ID

  • Message ID

Requirements to configure Syslog in Paragon Insights

Syslog ingest requires some setup before you can use it as a sensor in a rule:

  • Pattern - A pattern identifies some syslog event; you create a pattern for each event you want to monitor. You can configure patterns for both structured and unstructured events.

  • Pattern set - With the patterns configured, you then group them into a pattern set, which you then reference when defining the syslog sensor settings within a rule.

Before you configure Patterns and Pattern Set for Syslog ingest, note that the following fields are common in syslog messages. Paragon Insights extracts these fields and includes them automatically in the raw table, enabling you to make use of them directly when creating a rule, and avoiding the need to configure patterns.

To illustrate use of these values, consider the following example syslog messages:

Structured - <30>1 2019-11-22T03:17:53.605-08:00 R1 mib2d 28633 SNMP_TRAP_LINK_DOWN [junos@2636.1.1.1.2.29 snmp-interface-index="545" admin-status="up(1)" operational-status="down(2)" interface-name="ge-1/0/0.16"] ifIndex 545, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-1/0/0.16

Equivalent unstructured - <30>Nov 22 03:17:53 R1 mib2d[28633]: SNMP_TRAP_LINK_DOWN: ifIndex 545, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-1/0/0.16

System-generated fields:

  • "__log_priority__" - Priority of syslog message

    • In the examples, <30> denotes the priority

  • “__log_timestamp__" - Timestamp in epcoh in the syslog message

    • In the structured example, 2019-11-22T03:17:53.605-08:00 is converted to epoch with -08:00 indicating the time zone

    • In the unstructured example, the time zone from the configuration will be used to calculate epoch

  • "__log_host__" - Host name in the syslog message

    • In the examples, R1 denotes the host name

  • "__log_application_name__” - Application name in the syslog message

    • In the examples, mib2d is the application name

  • "__log_application_process_id__” - Application process ID in the syslog message

    • In the examples, 28633 is the ID

  • "__log_message_payload__" - Payload in the message

    • Structured example - “SNMP_TRAP_LINK_DOWN [junos@2636.1.1.1.2.29 snmp-interface-index="545" admin-status="up(1)" operational-status="down(2)" interface-name="ge-1/0/0.16"] ifIndex 545, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-1/0/0.16”

    • Unstructured example - “SNMP_TRAP_LINK_DOWN: ifIndex 545, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-1/0/0.16”

  • "Event-id" - Denotes the event ID configured in the pattern

    • In the examples, SNMP_TRAP_LINK_DOWN is the event ID

Note

Be sure not to define any new fields using a name already defined above.

To know how to configure Syslog ingest, see Configure Syslog Ingest.

iAgent (CLI/NETCONF)

For all the benefits of the ’push’ data collection methods, some operational and state information is available only through CLI/VTY commands. iAgent fills this gap by taking advantage of NETCONF/SSH functionality to provide HeathBot with the ability to connect to a device, run commands, and capture the output.

iAgent sensors use NETCONF/SSH and YAML-based PyEZ tables and views to fetch the necessary data. Both structured (XML) and unstructured (VTY commands and CLI output) data is supported.

With iAgent, the Paragon Insights server initiates SSH requests over any available network interface, whether in-band or out-of band; and the device responds (when properly configured) with the requested data.

In Paragon Automation Platform, the iAgent functionality is extended to other vendor devices from Arista Networks, Palo Alto Networks, and Cisco.

Using Netmiko, Paragon Insights makes SSH connections over the selected port to the third-party device. To gather device information, Paragon Insights sends CLI commands over SSH and receives string blobs back as output. The string blobs are then parsed through TextFSM, using ntc-templates into JSON format and then stored in the database. Default templates are located at /srv/salt/_textfsm. A repository of ntc-templates for network devices is available here: NTC Templates.

For advanced users who need a template which does not exist, you can create your own templates and upload them to Paragon Insights using the Upload Rule Files button on the Configuration > Rules page. User defined templates are stored at /jfit/_textfsm. The files must end with the .textfsm suffix. To know how to upload pre-defined rules in Paragon Automation Platform, see Add a Pre-Defined Rule.

You must configure a Junos device to send telemetry data using iAgent sensor or a NETCONF connection.

At minimum, iAgent (NETCONF) requires:

  • Junos OS Version: 11.4 or later

  • Minimum required device configuration:

    set system services netconf ssh

To configure iAgent or NETCONF port for a device added in Paragon Automation GUI, see Editable Fields on the Edit Devices Page table in Edit Devices.

TextFSM is integrated into PyEZ’s table/view feature which is an integral part of iAgent.

Example: PaloAlto Panos– Show Running Security Policy

To see the running security policy on a Panos device, we need to:

  • Define a table/view for it

  • Gather the output by sending the needed CLI to the device over SSH

  • Generate JSON to store in Paragon Insights database

Define PyEZ Table/View

We need to define a PyEZ table that is used by the iAgent rule assigned to the Panos device. The following table definition lacks a view definition. Because of this, the entire output from the show running security-policy ends up getting stored in the database after processing.

(Optional) To store only a portion of the received data in Paragon Insights, you can define a view in the same file. The view tells Paragon Insights which fields to pay attention to.

Gather Output from Device

Using an iAgent rule that references the PyEZ table (or table/view) defined above, Paragon Insights sends the command show running security-policy to the device which produces the following output:

Generate JSON for Use in Paragon Insights Database

Since the device configuration specifies Palo Alto Networks as the vendor and Panos OS as the operating system, the TextFSM template used for this example would look like this:

When the template above is used by Paragon Insights to parse the output shown previously, the resulting JSON looks like:

SNMP

SNMP is a widely known and accepted network management protocol that many network device manufacturers, including Juniper Networks, provide for use with their devices. It is a polling type protocol where network devices that are properly configured make configuration, diagnostic, and event information available to collectors, which must also be properly configured and authenticated. The collectors poll devices by sending specifically structured requests, called get requests, to retrieve data.

Paragon Insights supports SNMP as a sensor type, using standard get requests to gather statistics from the device. Paragon Insights makes requests over any available interface, whether in-band or out-of-band, and the device responds (when configured) with the requested data.

For information about SNMP as used on Junos OS devices, see Understanding SNMP Implementation in Junos OS.