ON THIS PAGE
NetFlow Collector Overview
NetFlow collector is a data collection tool in Paragon Pathfinder.
NetFlow
collector uses the netflowd microservice
in the
northstar namespace to collect and report data
about traffic flow in the network. Netflowd is automatically installed as part of the
Pathfinder installation package. Netflowd receives the NetFlow data from the routers,
decodes the records, and aggregates the data. Netflowd uses this aggregated data to
create demands, which indicate the amount of traffic flow in the network. Netflowd
stores the data in the Time Series Database (TSDB) and shares the data with the Path
Computation Server (PCS).
The
aggregated
data is
used to generate Demand reports that are available in Paragon
Pathfinder
(Reports
> Demand). These reports provide information on network traffic. This
data is also used in Paragon Planner to generate Demand reports, plan, and model the
network.
- The PCS monitors traffic from the autonomous systems (AS) and VPNs.
- The PCS supports both IPv4 and IPv6 traffic.
Pathfinder leverages the Junos OS implementation of flow monitoring and aggregation by using NetFlow Version 9 and Version 10 (IPFIX) flow templates. See the following Junos OS documentation for background:
- Configuring Flow Aggregation to Use Version 9 Flow Templates.
-
Configuring Flow Aggregation to Use IPFIX Flow Templates on MX, vMX and T Series Routers, EX Series Switches and NFX250.
-
Configuring Flow Aggregation to Use IPFIX Flow Templates on PTX Series Routers.
Demand Generation
Netflowd uses four aggregation keys (obtained from the NetFlow data that netflowd collects) to generate the demands:
- Ingress provider edge (PE) device (that is the device reporting the flow)
-
BGP next hop IP address
-
Routing table name
- When this key is present, it is the name of the VRF for which the ingress interface is configured.
- This key is absent if no VPN is associated with the demand. In this case, the ingress interface is configured in the default routing table.
- This key is displayed as NONE if netflowd is not able to determine whether the ingress interface is configured in the default routing table or on a VRF. That would happen, for example, if the PCS was not able to collect the snmp-indexes for the interfaces.
- Specification of IPv4 (displayed as IP in the Demand tab of the network information table) or IPv6
The values of the keys are indicated in the names of the demands which are displayed in the Name column of the Demand tab in the network information table. Here are some examples:
- vmx102_10.1.0.10/32_vpn100_IP
-
vmx102_10.1.0.10/32_IP (if no VPN is associated with the demand)
-
vmx102_10.1.0.10/32_NONE_IP (if it is unknown whether the ingress interface is configured on the default routing table or on a VRF)
NetFlow Collector Requirements
To use NetFlow collector in Pathfinder, you must:
- Install Nginx Ingress Controller when you install the Infrastructure component. See Install Multi-Node Cluster on CentOS and Install Multi-Node Cluster on Ubuntu.
-
Configure the network routers for flow monitoring (NetFlow v9 or v10). See Configuration on the Network Routers.
- Run the device collection task periodically to create and maintain an accurate VPN model in Pathfinder. We recommend that you run the device collection task at least once daily. See Add a Device Collection Task.
(Optional) Customize NetFlowd parameters from the CLI. See Customize Netflowd Parameters from the CLI.
Configuration on the Network Routers
To use NetFlow collector in Pathfinder, you must configure the network routers for flow monitoring (NetFlow v9 or v10) according to the router's operating system documentation.
Currently, only Juniper Networks devices and Cisco IOS-XR devices can be configured with NetFlow v9 and v10.
Here are some important considerations to keep in mind when you configure the routers:
- The NetFlow process (netflowd) identifies the device, which is reporting the
flow, through the source address (
inline-jflowstatement). Configure this parameter as the router’s loopback address. -
The flow-active-timeout parameter has a default value of 60 seconds. We recommend keeping it at 60 seconds or less.
- Configure the flow-server's IP address as the virtual IP (VIP) address that is configured for the Nginx Ingress Controller.
At the interface hierarchy level:
interfaces {
ge-0/0/1 {
unit 0 {
family inet {
sampling {
input;
}
address 10.0.21.1/24;
}
}
}
}At the forwarding-options hierarchy level:
forwarding-options {
sampling {
instance {
nfv9-ipv4 {
input {
rate 1;
run-length 0;
}
family inet {
output {
flow-inactive-timeout 15;
flow-active-timeout 60;
flow-server 172.16.18.1 {
port 9000;
version9 {
template {
nfv9-ipv4;
}
}
}
inline-jflow {
source-address 10.1.0.104;
}
}
}
} At the chassis hierarchy level:
chassis {
network-services enhanced-ip;
fpc 0 {
sampling-instance nfv9-ipv4;
}
} At the services hierarchy level:
services {
flow-monitoring {
version9 {
template nfv9-ipv4 {
nexthop-learning {
enable;
}
template-refresh-rate seconds 60;
option-refresh-rate seconds 60;
ipv4-template;
}
}
}
} At the interface hierarchy level:
interfaces {
ge-0/0/1 {
unit 0 {
family inet {
sampling {
input;
}
address 10.0.21.1/24;
}
}
}
}At the forwarding-options hierarchy level:
forwarding-options {
sampling {
instance {
nfv10-ipv4 {
input {
rate 1;
run-length 0;
}
family inet {
output {
flow-inactive-timeout 15;
flow-active-timeout 60;
flow-server 172.16.18.1 {
port 9000;
version-ipfix {
template {
nfv10-ipv4;
}
}
}
inline-jflow {
source-address 10.1.0.104;
}
}
}
}
}
}
} At the chassis hierarchy level:
chassis {
network-services enhanced-ip;
fpc 0 {
sampling-instance nfv10-ipv4;
}
} At the services hierarchy level:
services {
flow-monitoring {
version-ipfix {
template nfv10-ipv4 {
nexthop-learning {
enable;
}
template-refresh-rate {
seconds 60;
}
option-refresh-rate {
seconds 60;
}
ipv4-template;
}
}
}
} Customize Netflowd Parameters from the CLI
The parameters related to netflowd are configured by default and you can view these parameters from the CLI. Optionally, you can customize these parameters as well. Table 1 describes the netflowd parameters that you can customize from the CLI.
|
Parameter |
Command |
Command |
|---|---|---|
| enable-ssl | set northstar analytics netflowd enable-ssl | Configure this parameter to enable netflowd to establish a Secure Socket Layer (SSL) connection to the native datastore. |
| logging-parameters | set northstar analytics netflowd logging-parameters | Configure the level of information that is
captured in the log file. The default level is info. If you want more information to be included in the log file, you can set the level to debug. The log file will include all the flows which are received from each device, identified by the source IP address. You can also view, for each flow, all the fields that netflowd processes and parses. |
| default-sampling-interval |
set northstar analytics netflowd default-sampling-interval |
Configure the default sampling interval that is used if the router does not provide the interval in the Template FlowSet. Default: 1. |
| publish-interval |
set northstar analytics netflowd publish-interval |
Configure the interval (in seconds or minutes) to publish records to both the TSDB and the PCS. Traffic is aggregated per publishing interval. This value must be equal to or greater than the reporting time configured in the router (flow-active-timeout value) to ensure that for every publishing interval, all active flows are reported. Default: 60s. |
| notify-final-bandwidth-on-inactive-flow |
set northstar analytics netflowd notify-final-bandwidth-on-inactive-flow |
Configure this parameter to enable netflowd to send one final update after a flow is no longer active, reporting the bandwidth as 0. By default, this parameter is not configured. So, the bandwidth value is not reported once a flow becomes inactive; the last reported active value is the last value displayed. |
| aggregate-by-prefix |
set northstar analytics netflowd aggregate-by-prefix |
Configure this parameter to enable netflowd to aggregate all traffic from a specific ingress provider edge (PE) router to a specific destination (prefix) within the specified period. By default, NetFlow aggregates traffic by PE routers, but for some applications (such as Egress Peer Engineering and Ingress Peer Engineering), you would want the traffic to be aggregated by prefix. |
| stats-interval |
set northstar analytics netflowd stats-interval |
Configure the interval (in seconds) at which statistics are printed to the log file. By default, the interval is not configured, so the statistics are not printed to the log file. |
| generate-as-demands |
set northstar analytics netflowd generate-as-demands |
Configure this parameter to enable netflowd to generate AS demands. By default, this parameter is not configured. So, AS demands are not displayed through REST APIs or in Demand reports in the GUI, even if valid NetFlow records are being exported. |
| top-prefixes | set northstar analytics netflowd top-prefixes |
Configure the number of prefixes (in terms of the aggregated traffic volume) to be exported. Range: 1 through 10,000 |
| top-prefixes-export-ticks | set northstar analytics netflowd top-prefixes-export-ticks |
Configure the number of intervals above which traffic is aggregated for the top N prefixes, where the export interval length is determined by the publish-interval parameter. Example: If you set the publish-interval as 60s and top-prefixes-export-ticks as 5, the top N prefixes are exported (published) every 5 minutes (5x60s = 5m). |
| workers | set northstar analytics netflowd workers |
Configure the number of processes to be started. When set to 0, it takes the value of the number of cores in the system. Default: 1 |