Paragon Insights Push-Model Ingest Methods

Configure sFlow Ingest

As with other ingest methods, navigate to Settings > Ingest and choose the sFlow tab on the left of the Ingest window.

The Sflow Settings are broken down into 4 sections:

• Sample

  There are two pre-defined sample categories and each is represented in the sFlow header as an integer sample-type value. Table 2 below shows the sample types and their numeric value. Table 2: sFlow Sample Types Sample Type Integer Value in sFlow Header counter-sample 2 expanded-counter-sample 4 flow-sample 1 expanded-flow-sample 3 Note: The difference between the expanded sensor types and the non-expanded sample types is the size of the data fields. The field names and types are the same, but the field sizes are larger in the expanded sample types. Packet definitions for these sample types can be found here: sFlow Samples Table 3 shows the other fields contained in an sFlow sample header (by sample type) along with the field type. Table 3: sFlow Packet Header Fields field type/size in bits counter-sample flow-sample integer/32 sampleSequenceNumber sampleSequenceNumber integer/8 sourceIDType 0 = SNMP interface index 1 = VLAN ID (smonVlanDataSource) 2 = Physical entity (entPhysicalEntry) sourceIDType 0 = SNMP interface index 1 = VLAN ID (smonVlanDataSource) 2 = Physical entity (entPhysicalEntry) integer/24 sourceIDValue sourceIDValue integer/32 n (the number of sampled records contained in the Counter sample) sampleSamplingRate integer/32 - samplePool (number of packets that could have been sampled) integer/32 - sampleDroppedPackets (number of packets dropped due to lack of resources) integer/8 - sampleInputInterfaceFormat (input interface type) integer/32 - sampleInputInterfaceValue (input interface (SNMP interface index) integer/1 sampleOutputInterfaceFormat (output interface type) integer/33 - sampleOutputInterfaceValue (SNMP interface index) integer/32 - n (the number of flow records) data counter records flow records

• Flow Record

  The Flow Record section provides the tools needed to define the different types of flow that might be seen in an sFlow capture. Paragon Insights ships with 16 types of pre-defined flow records, each of which have a format number and a sensor path for use in defining sFlow rules, shown in Table 4 below. There are several fields in each type of flow record. These can be seen by selecting the desired record type from the list and clicking the edit (pencil) button. Table 4: Flow Record Types Record Type Format Number Sensor Path Value raw packet headers 1 /sflow-v5/flow-sample/raw-packet-header Ethernet frame data 2 /sflow-v5/flow-sample/ethernet-frame-data IPv4 data 3 /sflow-v5/flow-sample/ipv4-data IPv6 data 4 /sflow-v5/flow-sample/ipv6-data extended switch data 1001 /sflow-v5/flow-sample/extended-switch-data extended router data 1002 /sflow-v5/flow-sample/extended-router-data extended gateway data 1003 /sflow-v5/flow-sample/extended-gateway-data extended user data 1004 /sflow-v5/flow-sample/extended-user-data extended URL data 1005 /sflow-v5/flow-sample/extended-url-data extended MPLS data 1006 /sflow-v5/flow-sample/extended-mpls-data extended NAT data 1007 sflow-v5/flow-sample/extended-nat-data extended MPLS tunnel 1008 /sflow-v5/flow-sample/extended-mpls-tunnel extended MPLS VC 1009 /sflow-v5/flow-sample/extended-mpls-vc extended MPLS FEC 1010 /sflow-v5/flow-sample/extended-mpls-fec extended LVP FEC 1011 /sflow-v5/flow-sample/extended-mpls-lvp-fec extended VLAN tunnel 1012 /sflow-v5/flow-sample/extended-vlan-tunnel When you configure rules for sFlow, you can choose from any of these record types. You can create new flow records by clicking the add (+) icon on the Sflow Settings page.

• Counter Record

  The Counter Record section provides the definition for the two pre-defined counter record types. There are two types of counter records, ethernet-interface-counters and generic-interface-counters. Generic interface counters are format number 1 and Ethernet interface counters are format number 2. The sensor path for generic interface counters is /sflow-v5/counter-sample/generic-interface-counter. The sensor path for Ethernet interface counters is /sflow-v5/counter-sample/ethernet-interface-counter. The fields available within the counter records are the possible errors and the countable statistics such as: frame errors collisions deferred transmissions transmit errors administration status operational status input packets output packets input errors output errors and others

  You can use either the generic interface counter or Ethernet interface counter in rules that you define. The counter sensors can be defined to pick even single fields from either of the available counters. You can create additional counter record types by clicking the add (+) icon on the Sflow Settings page (Counter Record section).

• Protocol

  The Protocol section provides a means to define which protocol the sFlow captures contain and allow for the decoding of many network protocols. The fields that are contained in each protocol entry are the same fields as would be seen in a frame or packet of that type. For example, an Ethernet frame would have a destination MAC address, a source MAC address, and an ethernet-next-header-type field. The fields defined in any protocol you want to decode must appear in the protocol definition in the same order as they would appear in the packet or frame. The number column that appears is the IANA protocol number assigned to that protocol. For example, the tcp protocol is protocol number 6.

Extend sFlow Decoding Capabilities

You can extend the protocol decoding capabilities of Paragon Insights by creating additional decoding schemas under the Sample, Flow Record, Counter Record, and Protocol tabs on the sFlow ingest settings page.

Once you have updated the ingest settings, the new rules can make use of them as follows:

• Use a packet capture utility such as Wireshark to identify the order in which the new protocols appear within the sFlow packets.

• Ensure that the sensor path you use in the sensor definition is formatted like this: /sflow-v5/<sample-name>/<record-name>/<l2-protocol>/<l3-protocol>, where sample-name is the new sample, record-name is the new counter or flow record, and the l2-protocol and l3-protocol are the new protocols. Again, these protocols must be named and positioned as they appear in the sFlow captures.

Configure sFlow Rules

As with other rule definitions, sFlow rules are made up of sensors, fields, vectors, and so on. An sFlow sensor has a Sensor Name, a Sensor Type of sFlow, and an sFlow Path as shown in Figure 3.

Figure 3: sFlow Sensor Definition

The sensor path serves a big role in sensor definition. Paragon Insights uses the sensor path to define not only the sFlow flow type, but the sample type, record type, protocol, and other custom path elements if needed.

Delete sFlow Settings

To delete an sFlow Setting:

1. Click Settings > Ingest from the left-nav bar.

   The Ingest Settings page is displayed.

2. Click the sFlow tab to view the sFlow Settings page.

3. Do one of the following.

   • To delete an sFlow sample:

     1. Click Sample to view the list of sFlow samples.

     2. Select the sFlow sample that you want to delete.

     3. Click the delete (trash can) icon.

        The CONFIRM DELETE SAMPLE pop-up appears.

        Figure 4: Confirm Delete Sample Pop-up

   • To delete a flow record:

     1. Click Flow Record to view the list of flow records.

     2. Select the flow record that you want to delete.

     3. Click the delete (trash can) icon.

        The CONFIRM DELETE FLOW RECORD pop-up appears.

        Figure 5: Confirm Delete Flow Record Pop-up

   • To delete a counter record:

     1. Click Counter Record to view the list of counter records.

     2. Select the counter record that you want to delete.

     3. Click the delete (trash can) icon.

        The CONFIRM DELETE COUNTER RECORD pop-up appears.

        Figure 6: Confirm Delete Pop-up

   • To delete a protocol:

     1. Click Protocol to view the list of protocols.

     2. Select the protocol that you want to delete.

     3. Click the delete (trash can) icon.

        The CONFIRM DELETE PROTOCOL pop-up appears.

        Figure 7: Confirm Delete Protocol Pop-up

4. In the pop-up that appears, do any one of the following:

   • Click Yes to delete the sFlow setting from the database. However, the changes are not applied to the ingest service.

     Note:

     • We recommended that you do not delete an sFlow setting that is currently in use.

     • After you delete an sFlow setting from the database, you cannot configure that sFlow setting in new devices or device groups even if you have not deployed changes.

     • You can also deploy changes to the ingest service or roll back the changes that you have already deleted, from the PENDING CONFIGURATION page. For more information, see Commit or Roll Back Configuration Changes in Paragon Insights.

   • Select the Deploy changes check box and then click Yes to delete the sFlow setting from the database, and to apply the changes to the ingest service.

   • (Optional) Click No to cancel this operation.

Configure Syslog Ports

By default, Paragon Insights listens for syslog messages from all device groups on UDP port 514. You can change the system-level syslog port, as well as configure one or more ports per device group. The more specific device group setting takes precedence over the system level setting.

To change the system-level syslog port:

1. Click Settings > Ingest in the left-nav bar.

2. Select the Syslog tab on the Ingest Settings page.

3. On the Syslog page, edit the port number.

4. Click Save & Deploy.

To configure a syslog port for a device group:

1. Go to the Configuration > Device Group page and click on the name of a device group.

2. Click the Edit (pencil) icon.

3. In the pop-up window, enter the port(s) in the Syslog Ports field.

4. Click Save & Deploy.

Configure Time Zone for a Device

When a device exports structured syslog messages, time zone information is included within the message. However, unstructured syslog messages do not include time zone information. By default, Paragon Insights uses GMT as the time zone for a device. In these cases you can assign a time zone to a device or device group within Paragon Insights.

To configure a device’s time zone at the device group level:

1. Go to the Configuration > Device Group page and click on the name of a device group.

2. Click the Edit (pencil) icon.

3. In the pop-up window, enter a value in the Timezone field, for example -05:00.

To configure a device’s time zone at the device level:

1. Go to the Configuration > Device page and click on the name of a device.

2. Click the Edit (pencil) icon.

3. In the pop-up window, enter a value in the Timezone field, for example -05:00.

Configure Multiple Source IP Addresses for a Device

In cases where syslog messages arrive from a device using a different source IP address than the one originally configured in the Paragon Insights GUI, you can add additional source IP addresses.

To support additional source IP addresses:

1. Go to the Configuration > Device page and click on the name of a device.

2. Click the Edit (pencil) icon.

3. In the pop-up window, enter the IP address(es) in the Syslog Source IPs field.

Configure Host Name Aliases for a Device

When a device has more than one host name, such as a device with dual REs, syslog messages can arrive at the Paragon Insights server with a host name that is not the device’s main host name. In these cases, you can add host name aliases for that device.

To configure additional hostname aliases:

1. Go to the Configuration > Device page and click on the name of a device.

2. Click the Edit (pencil) icon.

3. In the pop-up window, enter the host name(s) in the Syslog Host Names field.

Syslog - Device Configuration

Syslog requires that you have at least the following configuration elements committed on your device(s) in addition to

• Junos OS Version: Any release

• Required configuration:

  ## 10.1x.x0.1 = Paragon Insights server

Add Syslog Configuration to the Network Device

If not already done, configure your network device(s) to send syslog data to Paragon Insights. The example device configuration from the previous section is repeated here:

## 10.1x.x0.1 = Paragon Insights server

Configure Events Patterns

An Event pattern is a configuration to monitor some syslog event; you create a pattern for each event you want to monitor. This example uses patterns to monitor four syslog events (two structured and two unstructured).

1. In the Paragon Insights GUI, click Settings > Ingest in the left-nav bar.

2. Select the Syslog tab on the Ingest Settings page.

3. On the Syslog page, click the plus (+) icon.

4. In the pop-up window that appears, enter the following parameters for the first pattern, named snmp-if-link-down:

   

5. Click Save and Deploy.

6. Click the plus (+) icon once more and enter the following parameters for the second pattern, named fpc-offline:

   
   Note:

   The full value entered in the Filter field is fpc%{NUMBER:fpc} Marking ports %{WORD:port-status}

7. Click Save and Deploy. On the Syslog page you should see the two patterns you just created.

Usage notes for the patterns

For structured syslog:

• The event ID (SNMP_TRAP_LINK_DOWN) references the event name found within the syslog messages.

• Fields are optional for structured syslog messages; if you don’t configure fields, the attribute names from the message will be treated as field names.

  • In this example, however we have user-defined fields:

    • The field names (if-name, snmp-index) are user-defined.

    • The field interface-name value is an attribute from the syslog message, for example, ge-0/3/1.0; this field is renamed as if-name

    • The field snmp-interface-index value is an attribute from the syslog message, for example, ifIndex 539; this field is renamed as snmp-index

    • The field snmp-interface-index here is defined as an integer; by default the fields extracted from a syslog message are of type string, however type integer changes this to treat the value as an integer

• The constant section is optional, in this example, we have user-defined constants.

  • The constant name ifOperStatus is user-defined; in this case it has the integer value of '2'

• Filter configuration is optional for a structured syslog, though you can do so if desired; if used, the filter-generated fields will override the fields included in the syslog message.

• The key fields section is optional; by default the hostname and event ID will be the keys used by Paragon Insights; add additional key fields here; in this example, we have key-fields, namely interface-name, where the name and value are extracted from the syslog message’s attribute-value pair

For unstructured syslog:

• The event ID is user defined, this case PSEUDO_FPC_DOWN

  • For example, neither the unstructured syslog Nov 22 02:27:05 R1 fpc1 Marking ports down nor its structured counterpart <166>1 2019-11-22T02:38:23.132-08:00 R1 - - - - fpc1 Marking ports down includes an event ID

• A filter must be used to derive fields (unlike proper structured syslog); this example uses fpc%{NUMBER:fpc} Marking ports %{WORD:port-status}, where fpc becomes the field name and NUMBER denotes the syntax used to extract the characters out of that particular portion of the message, for example “2”

  • An example of a syslog message that matches the grok filter is “fpc2 Marking ports down”

• constant fpc-status - has a string value of ‘online’

Regarding filters:

• By default in a pattern, field and constant values are a string; to treat it as an integer or float, define the pattern’s field type as integer or float

• For unstructured patterns, you must configure a filter as the messages are sent essentially as plain text and don’t include field info on their own

• Filters should always be written to match the portion of message after the event ID; this allows the filter to parse a syslog message irrespective of whether it arrives in unstructured or structured format

  • For example, the filter fpc%{NUMBER:fpc} Marking ports %{WORD:port-status} matches both versions of the following syslog message:

    • Structured: <166>1 2019-11-22T02:38:23.132-08:00 R1 - - - - fpc1 Marking ports down

    • Unstructured: Nov 22 02:27:05 R1 fpc1 Marking ports down

Clone an Existing Syslog Events Pattern

Starting with Paragon Insights Release 4.0.0, you can clone an existing Syslog pattern.

To clone an existing Syslog events pattern:

1. Click Settings > Ingest in the left-nav bar.

   The Ingest Settings page is displayed.

2. Click the Syslog tab to view the Syslog page.

3. Click to expand the Events Pattern section in the Syslog Settings page to view existing syslog events patterns.

4. To clone a pattern for Paragon Insights Release 4.0.0 and 4.1.0, click the Clone icon.

   To clone a particular template for Paragon Insights Release 4.2.0 and later, select the option button next to the name of the template and click Clone.

   The Clone Pattern: <name of syslog pattern> page is displayed.

   From the Clone Pattern: <name of syslog pattern> page, you can

   • Edit existing fields

   • Add new fields or constants

   • Add or remove key fields

5. Click Save to save configuration and clone the syslog pattern.

   Alternatively, click Save & Deploy to save configuration, clone syslog pattern, and deploy the pattern.

Add Patterns to a Pattern Set

With the patterns configured, group them into a pattern set.

1. On the Syslog page,click to expand the Pattern Set section and click the plus (+) icon.

2. In the pop-up window that appears, enter the following parameters:

   

3. Click Save and Deploy. On the Syslog page you should see the pattern set you just created.

Clone an Existing Syslog Pattern Set

Starting with Paragon Insights Release 4.0.0, you can clone an existing Syslog pattern set.

To clone an existing Syslog pattern set:

1. Click Settings > Ingest in the left-nav bar.

   The Ingest Settings page is displayed.

2. Click the Syslog tab to view the Syslog page.

3. Click to expand the Pattern Set section in the Syslog page to view existing syslog pattern sets.

4. To clone a pattern set for Paragon Insights Release 4.0.0 and 4.1.0, click the Clone icon.

   To clone a particular template for Paragon Insights Release 4.2.0 and later, select the option button next to the name of the template and click Clone.

   The Clone Pattern-set: <name of pattern-set> page is displayed.

   From the Clone Pattern-set: <name of pattern-set> page, you can

   • Edit the name and description fields

   • Add or remove patterns from the Patterns field.

5. Click Save to save configuration and clone the syslog pattern set.

   Alternatively, click Save & Deploy to save configuration, clone syslog pattern set, and deploy the pattern set.

Configure Header Pattern

Starting in Paragon Insights Release 4.0.0, you can configure the pattern for parsing the header portion of a syslog message. With this release, unstructured syslog messages of non-Juniper devices are supported. In earlier releases, you can only parse the payload portion of either a structured syslog message as specified in RFC 5424 standard, or a Juniper device’s unstructured syslog message.

In general, it is assumed that any unstructured syslog message matches the Juniper syslog message pattern. For example, you do not have to configure a Juniper header pattern as this pattern is inbuilt with Paragon Insights. However, in case of a non-Juniper device’s unstructured syslog message that does not match with the inbuilt pattern, a first match is made with one of the user-configured header patterns. Following a successful match, the fields are extracted. When there is no match, the incoming syslog message is dropped.

To configure a header pattern:

1. Navigate to Settings > Ingest in the left-nav bar.

   The Ingest Settings page is displayed.

2. Click Syslog to view the Syslog page.

3. Click to expand the Header Pattern section.

   The Header Pattern section of the Syslog page is displayed. You can add a new header pattern and edit or delete an existing header pattern from this page.

4. Click the plus (+) icon to add a new header.

   The Add Header Pattern page is displayed.

5. Enter the following information in the Add Header Pattern page.

   1. Enter a name for the header pattern in the Name field.

   2. Enter a description for the header pattern in the Description field.

      For example, you can provide a one-line description of why you are creating this header pattern.

   3. Enter the filter or regular expression (regex) for the header patter in the Filter field.

      Note:

      You can use regex101.com to edit, validate, and modify the filter pattern you want to add to the header pattern.

      An example of a filter pattern is (.*):([A-Z][a-z]{2} \d{1,2} \d{1,2}:\d{1,2}:\d{1,2}\.\d*)\s:\s([a-z]*)\[(\d*)\]:\s*(.*)\s*.

   4. log-host, log-timestamp, and log-message-payload of the Fields section are mandatory fields that determine the position of the header.

      In the Fields section,

      1. Click log-host, and enter the following information.

         • Enter a name for the log host in the Name field.

           log-host is the default name.

         • Enter a description for log-host in the Description field.

           The default description is Position of host name.

         • Enter the capture group value with prefix $ in the From field.

           The capture group determines from which position in the header the log-host starts.

      2. Click log-timestamp, and enter the following information.

         • Enter a name for the log timestamp in the Name field.

           log-timestamp is the default name.

         • Enter a description for log-timestamp in the Description field.

           The default description is Position of time stamp.

         • Enter the capture group value with prefix $ in the From field.

           The capture group determines from which position in the header the log-timestamp starts.

         Note:

         Ensure that timestamp format follows this sample timestamp format: “Jan _2 15:04:05 2006”. Otherwise parsing of syslog messages will lead to an undefined behaviour.

      3. Click log-message-payload, and enter the following information.

         • Enter a name for the log message payload in the Name field.

           log-message-payload is the default name.

         • Enter a description for log-message-payload in the Description field.

           The default description is Position of payload.

         • Enter the capture group value with prefix $ in the From field.

           The capture group determines from which position in the header the log-message-payload starts.

      4. (Optional) Click the plus (+) icon to add a new field.

         • Enter a name for the new field in the Name field.

         • Enter a description for the new field in the Description field.

         • Enter the capture group value with prefix $ in the From field.

           The capture group determines from which position in the header the new field starts.

         You can add one or more than one fields by clicking the plus (+) icon.

6. Enter the name(s) of key fields in the Key Fields text box.

7. Click Save to save configuration, or click Save & Deploy to save and immediately deploy the configuration.

   Alternatively, to cancel the configuration, click Cancel.

Editing an Existing Header Pattern

To edit an already configured header pattern:

1. Navigate to Settings > Ingest in the left-nav bar.

   The Ingest Settings page is displayed.

2. Click Syslog to view the Syslog page.

3. Click to expand the Header Pattern section.

   The Header Pattern section of the Syslog Setting page is displayed.

4. Select the header pattern you want to edit by selecting the check box next to the name of the header pattern, and click the Edit (pencil) icon.

   The Edit Header Pattern Header Name page is displayed.

5. After you have edited the required fields, click Save to save configuration.

   You can also click Save & Deploy to save and immediately deploy the edited configuration.

Configure a Rule Using the Syslog Sensor

With the syslog ingest settings complete, you can now create a rule using syslog as the sensor.

This rule includes three elements:

• A syslog sensor

• Four fields capturing data of interest

• A trigger that indicates when the interface goes down

1. Click Configuration > Rules in the left-nav bar.

2. On the Rules page, click the + Add Rule button.

3. On the page that appears, in the top row of the rule window, set the rule name. In this example, it is check-interface-status.

4. Add a description and synopsis if you wish.

5. Click the + Add sensor button and enter the following parameters in the Sensors tab:

   

6. Now move to the Fields tab, click the + Add field button, and enter the following parameters to configure the first field, named event-id:

   

7. Click the + Add field button again and enter the following parameters to configure the second field, named fpc-slot:

   

8. Click the + Add field button again and enter the following parameters to configure the third field, named if-name:

   

9. Click the + Add field button once more and enter the following parameters to configure the fourth field, named snmp-index:

   

10. Now move to the Triggers tab, click the + Add trigger button, and enter the following parameters to configure a trigger named link-down:

    

11. At the upper right of the window, click the + Save & Deploy button.

Usage Notes for the rule

• Sensor tab

  • The sensor name if-status-sensor is user-defined

  • The sensor type is syslog

  • Pattern set check-interface-status - reference to the pattern set configured earlier

  • If not set, the Maximum hold period defaults to 1s

• Fields tab

  • Four fields are defined; although the patterns are capturing more than four fields of data, this example defines four fields of interest here; these fields are used in the trigger settings

  • The field names (event-id, fpc-slot, if-name, snmp-index) are user-defined

  • path event-id - default field created by syslog ingest in the raw table; references the field from the pattern configuration

  • path fpc - references the value from the filter used in the unstructured pattern configuration

  • path if-name - references the field from the pattern configuration

    • Data if missing all interfaces - if the if-name value is not included in the syslog message, use the string value “all interfaces”

  • path snmp-index - references the field from the pattern configuration

• Triggers tab

  • The trigger name link-down is user-defined

  • frequency 2s - Paragon Insights checks for link-down syslog messages every 2 seconds

  • term is-link-down - when $event-id is like SNMP_TRAP_LINK_DOWN, in any syslog message in the last 300 seconds, make red and show the message Link down for $if-name(snmp-id: $snmp-index)

    • $event-id - $ indicates to reference the rule field event-id

    • Link down for $if-name(snmp-id: $snmp-index) - for example, “Link down for ge-2/0/0 of FPC 2”

    • $if-name - references the field value, i.e., the name of the interface in the syslog message

  • term is-fpc-down - when $event-id is like PSEUDO_FPC_DOWN, in any syslog message in the last 300 seconds, make red and show the message Link down for $if-name of FPC$fpc-slot

    • $event-id - $ indicates to reference the rule field event-id

    • $if-name - “all interfaces”

    • Link down for $if-name of FPC$fpc-slot - for example, “Link down for all interfaces of FPC 2”

Add the Rule to a Playbook

With the rule created, you can now add it to a playbook. For this example, create a new playbook to hold the new rule.

1. Click Configuration > Playbooks in the left-nav bar.

2. On the Playbooks page, click the + Create Playbook button.

3. On the page that appears, enter the following parameters:

   

4. Click Save & Deploy.

Apply the Playbook to a Device Group

To make use of the playbook, apply it to a device group.

1. On the Playbooks page, click the Apply (Airplane) icon for the check-interface-status playbook.

2. On the page that appears:

   • Enter a name

   • Select the desired device group

   • Click Run Instance

3. On the Playbooks page, confirm that the playbook instance is running. Note that the playbook may take some time to activate.