Contrail Insights JTI (UDP) Monitoring
Configure JTI Device
Contrail Insights supports UDP-based Junos Telemetry Interface (JTI) from network devices. With network devices supporting UDP-based JTI, Contrail Insights is able to stream data from the devices.
When configuring JTI devices, you can select all the sensors that need to be monitored. Using the required and optional configuration parameters that you input in the Configure Network Device page, Contrail Insights will push the configuration to the device and enable the device to stream data to collectors.
To configure a JTI device:
Select Settings in the top right of the Dashboard, then select Network Devices.
Click +Add Device and complete the configuration parameter fields. See Figure 1.
Figure 1: JTI Configuration Parameters in Configure Network Device Page
To allow Contrail Insights to configure the network device, have the following settings on your device and supply the device
usernameandpassword:set system services netconf ssh
Following is an example configuration that Contrail Insights adds on the device:
streaming-server appformix-telemetry { remote-address x.x.x.x; # collector ip, Contrail Insights will automatically assign the collector remote-port 42596; } export-profile appformix { local-address y.y.y.y; # Device local ip to send out data, need to be a revenue port local-port 21112; dscp 20; reporting-rate 60; format gpb; transport udp; } sensor test-sensor { server-name appformix-telemetry; export-name appformix; resource /junos/system/linecard/interface/; }In addition, you need to enable JTI plug-ins in your
group_vars/allto enable JTI monitoring in Contrail Insights and defineappformix_install_jti_dependencies:appformix_plugins: - { plugin_info: 'certified_plugins/jti_config_all_sensors.json' } appformix_install_jti_dependencies: true
JTI Monitoring Special Requirements
Traffic from JTI sensors is injected into the forwarding path, so the collector must be reachable by means of in-band connectivity. JTI sensor traffic does not get forwarded through the router’s management interface (for example, fxp0). Contrail Insights Collector in Figure 2 includes Contrail Insights Agent and network devices.
In Contrail Insights, you can edit ManagementIp and MetaData.JtiConfig.LocalAddress in
the device JSON file. If MetaData.JtiConfig.LocalAddress is not specified, Contrail Insights uses the ManagementIp as the device in-band IP setting in device. In addition, Contrail
Insights configures the device so that it streams its JTI data to
one of the appformix_network_agents nodes.
You can specify jti_inband_ip in
the Ansible inventory files to specify the in-band IP address of the
collector (server). See Figure 2.
[appformix_network_agents] 10.10.10.2 ansible_ssh_user='user' ansible_ssh_pass='pwd' jti_inband_ip='1.1.1.2'
If the jti_inband_ip is not specified
in the Ansible inventory file, Contrail Insights uses the hostname
of the appformix_network_agents node.
JTI Out of Band Configuration
Contrail Insights configures the devices properly based on user input including sensor name, sensor path, collector IP address, and device source IP address.
In some scenarios, user does not want to share credentials with Contrail Insights. As a result, Contrail Insights does not have the device credentials to configure the devices. Alternatively, you can use out of band JTI configuration scripts in SDK instead. Contrail Insights will discover all JTI network devices in your environment and push configurations to your devices using the script. This script only works when you have only one JTI collector in your setup.
Example out_of_band_jti_configuration.py script:
from jnpr.junos import Device
from jnpr.junos.utils.config import Config
import sys
import rest
import json
import os
# 1) This script runs inside appformix-controller container.
# 2) It assumes that appformix_token.rst file is present in the current directory
# 3) It assumes that NETCONF user and password is supplied as arg1, arg2 for
# the script and netconf ssh port as arg4
# 4) It takes collector inband ip as a argument as arg3. It assumes that there
# is only one collector for JTI.
# TODO: Read JTI distribution map from plugin definition, read jti_inband_ip
# from server definition and assign the devices to its correct collector. The
# blocking item here is we don't have v2 API for plugin definition
with open('appformix_token.rst') as json_file:
data = json.load(json_file)
APPFORMIX_MASTER_TOKEN = data['Token']['TokenId']
DEVICE_NETCONF_USERNAME = sys.argv[1]
DEVICE_NETCONF_PASSWORD = sys.argv[2]
# jti_inband_ip of appformix_platform
APPFORMIX_CONFIG_COLLECTOR_DATA_IP = sys.argv[3]
NETCONF_PORT = sys.argv[4]
# You can change the following parameters based on requirement
LOCAL_PORT = '21112'
PAYLOAD_SIZE = '5000'
APPFORMIX_JTI_LISTEN_PORT = '42596'
HEADERS = {'content-type': 'application/json',
'X-Auth-Type': 'appformix',
'X-Auth-Token': APPFORMIX_MASTER_TOKEN}
url = 'http://localhost:80/appformix/controller/v2.0/network_devices'
resp = rest.get(url=url, headers=HEADERS)
result = json.loads(resp.text)
devices = []
for entry in result['NetworkDeviceProfile']:
if 'user.jti' in entry['NetworkDevice']['Source']:
device_config = {'ip': entry['NetworkDevice']['ManagementIp'],
'sensor_list':
entry['NetworkDevice']['MetaData']['JtiConfig']['SensorList'],
'device_data_ip':
entry['NetworkDevice']['MetaData']['JtiConfig']['LocalAddress'],
'report_rate':
entry['NetworkDevice']['MetaData']['JtiConfig']['ReportRate']}
devices.append(device_config)
for entry in devices:
# Create a Device Object
print "Connecting to device {}".format(entry['ip'])
dev = Device(host=entry['ip'],
user=DEVICE_NETCONF_USERNAME,
password=DEVICE_NETCONF_PASSWORD,
port=NETCONF_PORT)
try:
dev.open()
cu = Config(dev)
except Exception as e:
print "Fail to connect to device {}: {}".format(
entry['ip'], e)
continue
print "Configuring the streaming-server in device"
# Update the streaming-server, update the collector' in_band ip
msg = ("set services analytics streaming-server " +
"appformix-telemetry remote-address {} remote-port {}").format(
APPFORMIX_CONFIG_COLLECTOR_DATA_IP, APPFORMIX_JTI_LISTEN_PORT)
cu.load(msg, format='set')
print "Configuring the export-profile in device"
# Update the analytics export-profile, update the device's in_band ip
msg = ("set services analytics export-profile appformix " +
"local-address {}").format(entry['device_data_ip'])
cu.load(msg, format='set')
msg = ("set services analytics export-profile appformix " +
"transport udp format gpb reporting-rate {} " +
"local-port {} payload-size {}")
msg = msg.format(entry['report_rate'], LOCAL_PORT, PAYLOAD_SIZE)
cu.load(msg, format='set')
# Commit the change to device, rollback if commit fail
try:
cu.commit()
except Exception as e:
print "Fail to configure device {}".format(e)
cu.rollback()
continue
# Add sensor to the device
for sensor in entry['sensor_list']:
print "Configuring the sensor {} in device".format(sensor['Resource'])
msg = ("set services analytics sensor {} resource {} " +
"export-name appformix server-name appformix-telemetry")
msg = msg.format(sensor['Name'], sensor['Resource'])
cu.load(msg, format='set')
try:
cu.commit()
except Exception as e:
print "Fail to configure device sensor {}".format(e)
cu.rollback()
dev.close()
print "Closing connection to device {}".format(entry['ip'])
Troubleshooting
On the Contrail Insights Platform host, check if the Agent is listening on UDP port 42596 by running the following command.
netstat -lanp | grep 42596
If not, check if plug-in is posted. Check the
jti_network_deviceplug-in fromplugin_definitionendpoint in the Contrail Insights Platform API to see if the distribution_map in Config > ObjectList is correct.Check the network device configuration. On the device, from the CLI Configuration mode, running
show service analyticsshould have:A streaming server named "appformix-telemetry”
An export profile named “appformix”
And a sensor named “Interface_Sensor”
If any of these items are missing, look at the following file and check the log for authentication failures.
/var/log/appformix/controller/appformix/appformix_celery_queue_server_worker_celery.log
Check if data is being received at Contrail Insights Platform host. Run
tcpdumpto check if data is received by the Contrail Insights Platform host on UDP port 42596. If data is not being received from the network device on UDP port 42596, then it is likely that the in-band connectivity is not working. Thelocal-addressconfigured in streaming server "appformix-telemetry" must be able to reach the Contrail Insights Platform host address configured in the export profile.Check if data is being dropped by kernel. Following is an example output of
tcpdump:root@ubuntu:/home/acelio# tcpdump -nli p1p1 port 42596 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on p1p1, link-type EN10MB (Ethernet), capture size 65535 bytes 14:18:32.373370 IP 10.87.68.120.21112 > 10.87.68.13.42596: UDP, length 2320
If your output is similar to the following example, it indicates AppFormix-VM is dropping packets coming from the device, which can be a maximum transmission unit (MTU) issue:
root@ubuntu:~# tcpdump -nli eth0 port 42596 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes 16:28:25.165580 IP 10.27.73.254.21112 > 10.27.73.155.42596: UDP, bad length 3245 > 1472
If you are using CentOS or Red Hat software, check your IPtables rules if they block the traffic. You can run the following commands to remove IPtables rules in your AppFormix-VM:
iptables --flush iptables -P FORWARD ACCEPT iptables -P OUTPUT ACCEPT iptables -P INPUT ACCEPT service iptables save
These commands will remove all IPtables rules blocking the traffic and add rules accepting traffic.
You might also need to disable
rp_filteron the collector side:echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter echo 0 > /proc/sys/net/ipv4/conf/{jti_interface_name}/rp_filterFurther debugging can be done using the following script bundled with Contrail Insights. This script should be run on the Contrail Insights Agent that is monitoring the affected network device:
cd /opt/appformix/manager/tailwind/manager/ source ../ven/bin/activate python check_jti_device_test.py
This script will print out data if Contrail Insights receives JTI messages from the socket. If you do see tcpdump in the port 42596 but no data from this script, it means message has been dropped by the kernel.
Packages Needed for JTI Network Device Monitoring
Currently, you need to specify on which Agents JTI network devices should stream their metrics to. On those Contrail Insights Agents, you need to install the following three packages:
sudo apt-get install netcat sudo apt-get install protobuf-compiler sudo apt-get install libprotobuf-dev
These packages are needed for receiving and decoding JTI messages.