NetFlow Ingest

For the sake of completeness and to ensure correct context, the overview information from HealthBot Concepts is repeated here.

How NetFlow Works

NetFlow is a network protocol for collecting IP traffic statistics, which can then be exported to a tool for analysis. NetFlow is available in different versions, the latest being NetFlow v9 and NetFlow v10. The NetFlow v9 data export format is described in RFC 3954; NetFlow v10 is officially known as IPFIX and standardized in RFC 7011.

Junos devices support flow monitoring and aggregation using these protocols. The Junos OS samples the traffic, builds a flow table, and sends the details of the flow table over a configured UDP port to a collector, in this case Healthbot. HealthBot receives the incoming Netflow data, auto-detects it as v9 or v10, and process it further.

As shown above, the network device pushes data from the Packet Forwarding Engine, that is, directly from a line card. This means flow data is sent over the forwarding plane, so the collector must have in-band connectivity to the device. To use the flow sensor option, you configure the device with settings that include where to send the flow data. When you configure HealthBot to start collecting the data, the flow data is already flowing towards the server.

Flow Templates

Where other ingest methods have established sensor formats and identification details - for example, native GPB references paths, SNMP references MIBs, etc. - flow has no equivalent mechanism. Instead, HealthBot uses templates. These flow templates provide a mechanism to identify and decode incoming flow data before sending it for further processing.

HealthBot provides predefined flow templates for NetFlow v9 and v10 (IPFIX), or you can define your own. The predefined templates match those which the Junos OS currently supports. For example, the Junos OS template, ipv4-template, aligns with the HealthBot template hb-ipfix-ipv4-template. To view the fields used in the Junos OS templates, see Understanding Inline Active Flow Monitoring.

Flow Ingest Processing

The raw flow data that HealthBot receives is in binary format and unreadable. In order to make this data usable, HealthBot processes the incoming flow data as follows:

• HealthBot listens for incoming flow data on a configured port

• Since NetFlow messages don’t include a field that identifies the sending device, HealtBot uses the configured source IP address to derive a device ID

• Templates identify and decode incoming flow data to determine which fields it contains

• HealthBot then normalizes the fields for further use within the system

The resulting decoded and normalized data is now in a readable and usable format. Here is an example of flow data decoded using the hb-ipfix-ipv4-template template:

hb-ipfix-ipv4-template, destinationIPv4Address=192.168.48.200,destinationTransportPort=443,icmpTypeCodeIPv4=0,ingressInterface=1113, ipClassOfService=0,protocolIdentifier=6,sourceIPv4Address=172.16.235.191,sourceTransportPort=51032, bgpDestinationAsNumber=4200000000i,bgpSourceAsNumber=65000i,destinationIPv4PrefixLength=24i, dot1qCustomerVlanId=0i,dot1qVlanId=0i,egressInterface=1041i,flowEndMilliseconds=1483484591502u, flowEndReason=2i,flowStartMilliseconds=1483484577531u,ipNextHopIPv4Address="192.168.41.49", maximumTTL=120i,minimumTTL=120i,octetDeltaCount=188i,packetDeltaCount=4i,sourceIPv4PrefixLength=19i, tcpControlBits=16i,vlanId=0i 1483484642000000000

The data shows information from the NetFlow messages using naming according to IPFIX Information Elements. For example, destinationIPv4Address maps to element ID 12 in the elements table.

Configuration - Device Configuration

To use NetFlow as an ingest method in HealthBot, you must add configuration to the device you wish to monitor, to enable it to export flow data into HealthBot.

This example includes the Netflow v10 IPv4 template; adjust as needed for your environment. If not already done, complete device configuration to send NetFlow data to HealthBot as shown below.

Add the Device In HealthBot

1. In the HealthBot GUI, click Configuration > Device in the left-nav bar, and click the add device button (+ Device).
2. Click the + (Add Device) button
3. In the Add Device(s) window that appears, fill in the appropriate fields.

   Be sure to fill in the Flow IPs field with the IP address(es) from which NetFlow data will arrive.

4. Click Save & Deploy.

For more information about adding a device, see Adding a Device in Managing Devices, Device Groups, and Network Groups.

Usage Notes:

• Incoming NetFlow messages don’t include a device ID; HealthBot uses the message’s source IP address to derive a device ID

• When configuring this step, use the in-band interface IP address you configured in the sampling instance configuration on the device.

Add Device Group

1. Click Configuration > Device Group
2. Click the + (Add Device Group) button
3. In the Add Device Group window that appears, fill in the fields as appropriate. In the Flow Ports field, enter the port(s) on which NetFlow data will arrive.Note

   If your HealthBot installation is a multi-node installation using Kubernetes, you must also specify which HealthBot nodes will receive the NetFlow traffic by filling in the Flow Deploy Nodes field in the Add/Edit Device Group window.

4. Click Save & Deploy

   Usage Notes:

   • HealthBot will listen for NetFlow messages on this port for devices in this group.

   • The configured NetFlow ingest ports cannot be the same across device groups. You must configure a different port (or ports) for each group.

Define NetFlow Ingest Settings - Review Predefined Templates

1. Click Settings > Ingest Settings in the left-nav bar.
2. Click the Netflow tab on the left side of the page.
3. On the Netflow settings page, review the available templates for use in a rule.

Usage Notes:

• Notice that there are default flow templates for IPv4, IPv6, MPLS, MPLS-IPv4, MPLS-IPv6, and VPLS, for each of NetFlow v9 and v10.

• The flow templates include recognition patterns, called include fields and exclude fields, which help to recognize, identify, and categorize the incoming messages.

• Since NetFlow messages don’t distinguish between keys and values (all fields are simply incoming data), the templates specify which fields should be treated as keys for raw data.

Define NetFlow Ingest Settings - (Optional) Create Your Own NetFlow Template

1. On the Netflow settings page, click the + Template button.
2. In the Add Template window that appears, fill in the following fields (you can leave the other settings as is):

   • Template Name - give the template a name

   • NetFlow version - select v9 or v10

   • Priority - Available values are 1 through 10

   • Include Fields - add one or more fields that you want included in the template you wish to use

   • Exclude Fields - add one or more fields that you do not want included in the template you wish to use

   • Key Fields - specify which fields in the incoming messages should be treated as keys

3. Click Save & Deploy

   You should now see the template added to the NetFlow settings page.

4. (Optional) Repeat the steps above to create more templates.

Usage Notes:

• Priority - when a playbook includes multiple rules using the flow sensor, the priority value identifies which sensor and template gets priority over the other(s).

• Include/Exclude fields - include fields to help identify the template to use, or at least a ‘short list’ of templates to use; exclude fields then narrow down to the single desired template.

  • Example 1 - consider the hb-ipfix-ipv4-template template: it includes two IPv4 fields to narrow down to hb-ipfix-ipv4-template and hb-ipfix-mpls-ipv4-template, and excludes an MPLS field to eliminate hb-ipfix-mpls-ipv4-template, leaving only hb-ipfix-ipv4-template.

  • Example 2 - consider the hb-ipfix-mpls-ipv4-template template: it includes the same two IPv4 fields to narrow down to hb-ipfix-ipv4-template and hb-ipfix-mpls-ipv4-template. It also includes an MPLS field, which immediately eliminates the former template and leaving the latter as the template to use.

Configure a Rule Using the Flow Sensor

1. Click Configuration > Rules in the left-nav bar.
2. On the Rules page, click the + Add Rule button.

   The Rules page refreshes to show a nearly empty rule on the right part of the page.

3. In the top row of the rule window, leave the topic set as external and set the rule name that appears after the slash (/). In this example, it is periodic-aggregation-flow-rule.
4. Add a description and synopsis if you wish.
5. Click the + Add Sensor button and enter the following parameters in the Sensors tab:

   

6. Now move to the Fields tab, click the + Add Field button and enter the following parameters to configure the first field, source-ipv4-address:

   

7. Click the + Add Field button again and enter the following parameters to configure the second field, destination-ipv4-address:

   

8. Click the + Add Field button again and enter the following parameters to configure the third field, sensor-traffic-count:

   

9. Click the + Add Field button again and enter the following parameters to configure the fourth field, total-traffic-count:

   

10. Click the + Add Field button again and enter the following parameters to configure the fifth field, traffic-count-maximum:

    

11. Click the + Add Field button once more and enter the following parameters to configure the sixth field, traffic-count-minimum:

    

12. As the last step for the fields configuration, set the field aggregation time-range value to 10s:

    

13. Now move to the Variables tab, click the + ADD VARIABLE button and create the traffic-count-max and traffic-count-min variables that are the constants for the traffic-count-maximum and traffic-count-minimum fields, respectively.

    

    Note

    Only the definition for the traffic-count-max is shown graphically. Choose an appropriate Default Value when configuring both traffic-count-max and traffic-count-min variables. The value shown above is for testing purposes only and may not be appropriate for your network.

14. Now move to the Triggers tab, click the + Add trigger button and enter the following parameters to configure a trigger called traffic-measurement-trigger:

    

15. At the upper right of the window, click the Save & Deploy button.

Usage Notes:

• Sensor Tab:

  • The sensor name ipv4-flow-sensor is user-defined

  • The sensor type is flow

  • The sensor uses the predefined template hb-ipfix-ipv4-template

• Variables Tab:

  • The variables traffic-count-max and traffic-count-min are statically configured integers. In this case the values represent Bytes per second

  • These values are referenced in fields traffic-count-maximum and traffic-count-minimum and provide a reference point to compare against the total-traffic-count field

• Fields Tab:

  • Six fields are defined; some fields are used in the trigger settings while one field is referenced within another field

  • The field names are user-defined fields (UDF)

  • Fields source-ipv4-address, destination-ipv4-address, and sensor-traffic-count are extracting information from the flow sensor input

  • Path values for these fields identify specific values from the NetFlow messages, using naming according to IPFIX Information Elements

  • Fields source-ipv4-address and destination-ipv4-address have the Add to rule key setting enabled, indicating that this field should be shown as a searchable key for this rule on the device health pages

  • Field total-traffic-count - sums the IPv4 packet count from the sensor-traffic-count field every 10 seconds

  • The fields traffic-count-maximum and traffic-count-minimum are simply fixed values; the values are derived from the variables defined above

  • Field aggregation time-range - typically set to a value higher (longer) than individual field time range settings with the aim of reducing the frequency of information being sent to the database

• Triggers Tab:

  • The trigger name traffic-measurement-trigger is user-defined.

  • frequency 90s - HearthBot compares traffic counts every 90 seconds

  • In the term traffic-abnormal-gr:

    • When $total-traffic-count (the periodic count of incoming IPv4 traffic) is greater than $traffic-count-maximum (2500 Bps), show red and the message: “Total traffic count is above normal. Current total traffic count is : $total-traffic-count”.

  • In the term traffic-abnormal-ls:

    • When $total-traffic-count (the periodic count of incoming IPv4 traffic) is less than $traffic-count-minimum (500 Bps), show yellow and the message: “Total traffic count is below normal. Current total traffic count is : $total-traffic-count”.

  • In the term default-term:

  • Otherwise, show green and the message: “Total traffic count is normal. Current total traffic count is : $total-traffic-count”.

Add the Rule to a Playbook

1. Click Configuration > Playbooks in the left-nav bar.
2. On the Playbooks page, click the create + Create Playbook button.
3. On the page that appears, enter the following parameters:

   

4. Click Save & Deploy

Apply the Playbook to a Device Group

1. On the Playbooks page, click the Apply (Airplane) icon for the playbook you configured above.
2. On the Run Playbook page that appears

   • Enter a name for the playbook instance.

   • Select the desired device group from the Apply Group pull-down menu.

   • Click Run Instance.

3. On the Playbooks page, confirm that the playbook instance is running. Note that the playbook may take some time to activate.

Monitor the Devices

1. Click Monitor > Device Group Health in the left-nav bar and select the device group to which you applied the playbook from the Device Group pull-down menu.
2. Select one of more of the devices to monitor.
3. In the Tile View, the external tile contains the parameters from the rule you configured earlier.