Underlay Overlay Mapping in Contrail

Today’s cloud data centers consist of large collections of interconnected servers that provide computing and storage capacity to run a variety of applications. The servers are connected with redundant TOR switches, which in turn, are connected to spine routers. The cloud deployment is typically shared by multiple tenants, each of whom usually needs multiple isolated networks. Multiple isolated networks can be provided by overlay networks that are created by forming tunnels (for example, gre, ip-in-ip, mac-in-mac) over the underlay or physical connectivity.

As data flows in the overlay network, Contrail can provide statistics and visualization of the traffic in the underlay network.

Starting with Contrail Release 2.20, you can view a variety of analytics related to underlay and overlay traffic in the Contrail Web user interface. The following are some of the analytics that Contrail provides for statistics and visualization of overlay underlay traffic.

• View the topology of the underlay network.

  A user interface view of the physical underlay network with a drill down mechanism to show connected servers (contrail computes) and virtual machines on the servers.

• View the details of any element in the topology.

  You can view details of a pRouter, vRouter, or virtual machine link between two elements. You can also view traffic statistics in a graphical view corresponding to the selected element.

• View the underlay path of an overlay flow.

  Given an overlay flow, you can get the underlay path used for that flow and map the path in the topology view.

Accumulation of the data to map an overlay flow to its underlay path is performed in several steps across Contrail modules.

The following outlines the essential steps:

The contrail-snmp-collector and the contrail-topology are new daemons that are both added to the contrail-analytics node. The contrail-analytics package contains these new features and their associated files. The contrail-status displays the new services.

Example: contrail-status

The following is an example of using contrail-status to show the status of the new process and service for underlay overlay mapping.

Example: Service Command

The service command can be used to start, stop, and restart the new services. See the following example.

This section outlines the external interface configurations necessary for successful underlay overlay mapping for Contrail analytics.

The typical physical topology includes:

• Servers connected to the ToR switches.

• ToR switches connected to spine switches.

• Spine switches connected to core switches.

The following is an example of how the topology is depicted in the Contrail WebUI analytics.

Figure 1: Analytics Topology

Configure SNMP on the physical devices so that the contrail-snmp-collector can read SNMP data.

The following shows an example SNMP configuration from a Juniper Networks device.

set snmp community public authorization read-only

Configure LLDP on the physical device so that the contrail-snmp-collector can read the neighbor information of the routers.

The following is an example of LLDP configuration on a Juniper Networks device.

set protocols lldp interface all

set protocols lldp-med interface all

Flow samples are sent to the contrail-collector by the physical devices. Because the contrail-collector supports the sFlow and IPFIX protocols for receiving flow samples, the physical devices, such as MX Series devices or ToR switches, must be configured to send samples using one of those protocols.

Example: sFlow Configuration

The following shows a sample sFlow configuration. In the sample, the IP variable <source ip>refers to the loopback or IP that can be reachable of the device that acts as an sflow source, and the other IP variable <collector_IP_data> is the address of the collector device.

Example: IPFIX Configuration

The following is a sample IPFIX configuration from a Juniper Networks device. The IP address variable <ip_sflow collector> represents the sflow collector (control-collector analytics node) and <source ip> represents the source (outgoing) interface on the router/switch device used for sending flow data to the collector. This could also be the lo0 address, if it s reachable from the Contrail cluster.

Information about the physical devices must be sent to the SNMP collector before the full analytics information can be read and displayed. Typically, the pRouter information is taken from the contrail-config file.

SNMP collector getting pRouter information from contrail-config file

The physical routers are added to the contrail-config by using the Contrail user interface or by using direct API, by means of provisioning or other scripts. Once the configuration is in the contrail-config, the contrail-snmp-collector gets the physical router information from contrail-config. The SNMP collector uses this list and the other configuration parameters to perform SNMP queries and to populate pRouter UVEs.

Figure 2: Add Physical Router Window

pRouter UVEs are accessed from the REST APIs on your system from contrail-analytics-api, using a URL of the form:

http://<host ip>:8081/analytics/uves/prouters

The following is sample output from a pRouter REST API:

Figure 3: Sample Output From a pRouter REST API

Details of a pRouter UVE can be obtained from your system, using a URL of the following form:

http://<host ip>:8081/analytics/uves/prouter/a7-ex3?flat

The following is sample output of a pRouter UVE.

Figure 4: Sample Output From a pRouter UVE

The topology view and related functionality is accessed from the Contrail Web user interface, Monitor > Physical Topology.

To enable the Physical Topology section in the Contrail Web UI:

In the Contrail user interface, it is possible to drill down through displayed topology to the virtual machine level. The following diagram shows the virtual machines instantiated on a7s36 vRouter and the full physical topology related to each.

Figure 5: Physical Topology Related to a vRouter

At Monitor > Physical Topology, double click any link on the topology to display the traffic statistics graph for that link. The following is an example.

Figure 6: Traffic Statistics Graph

Click the Trace Flows tab to see a list of active flows. To see the path of a flow, click a flow in the active flows list, then click the Trace Flow button. The path taken in the underlay by the selected flow displays. The following is an example.

Figure 7: List of Active Flows

Limitations of Trace Flow Feature

Because the Trace Flow feature uses ip traceroute to determine the path between the two vRouters involved in the flow, it has the same limitations as the ip traceroute, including that Layer 2 routers in the path are not listed, and therefore do not appear in the topology.

Click the Search Flows tab to open a search dialog, then click the Search button to list the flows that match the search criteria. You can select a flow from the list and click Map Flow to display the underlay path taken by the selected flow in the topology. The following is an example.

Figure 8: Underlay Path

The schema to query the underlay mapping information for an overlay flow is obtained from a REST API, which can be accessed on your system using a URL of the following form:

http://<host ip>:8081/analytics/table/OverlayToUnderlayFlowMap/schema

Example: Overlay to Underlay Flow Map Schema

The schema for underlay data across pRouters is defined in the Contrail installation at:

http://<host ip>:8081/analytics/table/StatTable.UFlowData.flow/schema

Example: Flow Data Schema for Underlay

Example: Typical Query for Flow Map

The following is a typical query. Internally, the analytics-api performs a query into the FlowRecordTable, then into the StatTable.UFlowData.flow, to return list of (prouter, pifindex) pairs that give the underlay path taken for the given overlay flow.

The Contrail SNMP collector uses a Net-SNMP library to talk to a physical router or any SNMP agent. Upon receiving SNMP packets, the data is translated to the Python dictionary, and corresponding UVE objects are created. The UVE objects are then posted to the SNMP collector.

The SNMP module sleeps for some configurable period, then forks a collector process and waits for the process to complete. The collector process goes through a list of devices to be queried. For each device, it forks a greenlet task (Python coroutine), accumulates SNMP data, writes the summary to a JSON file, and exits. The parent process then reads the JSON file, creates UVEs, sends the UVEs to the collector, then goes to sleep again.

The pRouter UVE sent by the SNMP collector carries only the raw MIB information.

Example: pRouter Entry Carried in pRouter UVE

The definition below shows the pRouterEntry carried in the pRouterUVE. Additionally, an example LldpTable definition is shown.

The following create a virtual table as defined by:

The topology module reads UVEs posted by the SNMP collector and computes the neighbor table, populating the table with remote system name, local and remote interface names, the remote type (pRouter or vRouter) and local and remote ifindices. The topology module sleeps for a while, reads UVEs, then computes the neighbor table and posts the UVE to the collector.

The pRouter UVE sent by the topology module carries the neighbor list, so the clients can put together all of the pRouter neighbor lists to compute the full topology.

The corresponding pRouter UVE definition is the following.

An IPFIX and sFlow collector has been implemented in the Contrail collector. The collector receives the IPFIX and sFlow samples and stores them as statistics samples in the analytics database.

Example: IPFIX sFlow Collector Data

The following definition shows the data stored for the statistics samples and the indices that can be used to perform queries.

This section provides a variety of links where you can research errors that may occur with underlay overlay mapping.

System Logs

Logs for contrail-snmp-collector and contrail-topology are in the following locations on an installed Contrail system:

/var/log/contrail/contrail-snmp-collector-stdout.log

/var/log/contrail/contrail-topology.log

Introspect Utility

Use URLs of the following forms on your Contrail system to access the introspect utilities for SNMP data and for topology data.

• SNMP data introspect

  http://<host ip>:5920/Snh_SandeshUVECacheReq?x=PRouterEntry

• Topology data introspect

  http://<host ip>:5921/Snh_SandeshUVECacheReq?x=PRouterLinkEntry

The usual mechanism for adding pRouter objects to contrail-config is through Contrail UI. But you also have the ability to add these objects using the Contrail vnc-api. To add one pRouter, save the file with the name cfg-snmp.py, and then execute the command as shown:

python cfg-snmp.py

Example: Content for cfg-snmp.py