Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

 
 

Example: Troubleshoot a VXLAN Overlay Network with Overlay Ping and Overlay Traceroute on QFX Series Switches

In a Virtual Extensible LAN (VXLAN) overlay network, the existing ping and traceroute commands can verify the basic connectivity between two Juniper Networks devices that function as virtual tunnel endpoints (VTEPs) in the underlying physical network. However, in between the two VTEPs, there could be multiple routes through intermediary devices to the same destinations, and the ping and traceroute packets might successfully reach their destinations, while a connectivity issue exists in another route along which the data packets are typically forwarded.

With the introduction of the overlay parameter and other options in Junos OS Release 14.1X53-D30 for QFX5100 switches, you can use the ping and traceroute commands to troubleshoot a VXLAN overlay network.

For ping and traceroute mechanisms to work in a VXLAN overlay network, the ping and traceroute packets, also referred to as Operations, Administration, and Management (OAM) packets, must be encapsulated with the same VXLAN UDP headers (outer headers) as the data packets forwarded over the VXLAN segment with possible connectivity issues. If any connectivity issues arise, the overlay OAM packet would experience the same issues as the data packet.

This example shows how to use overlay ping and traceroute on a VTEP to verify the following in a VXLAN overlay network:

  • Scenario 1—Verify that a particular VXLAN is configured on another VTEP.

  • Scenario 2—Verify that the MAC address of a particular endpoint is associated with a VXLAN on another VTEP.

  • Scenario 3—Verify that no issues exist in a particular data flow between sending and receiving endpoints.

Note:

When issuing the ping overlay and traceroute overlay commands, the source VTEP on which you issue the command and the destination VTEP that receives the ping or traceroute packet must be Juniper Networks devices that support overlay ping and traceroute.

Requirements

This example uses the following hardware and software components:

  • Three physical (bare-metal) servers on which applications directly run.

  • Two QFX5100 switches running Junos OS Release 14.1X53-D30 or later software. These switches function as VTEPs.

  • Two Layer 3 routers, which can be Juniper Networks routers or routers provided by another vendor.

Before issuing the ping overlay and traceroute overlay commands, gather the information needed for each parameter—for example, IP addresses or MAC addresses—used for a particular scenario. See Table 1 to determine which parameters are used for each scenario.

Overview and Topology

The VXLAN overlay network topology shown in Figure 1 includes physical servers A, B, and C on which applications directly run. The applications on physical servers A and B need to communicate with the applications on physical server C. These servers are on the same subnet, so the communication between the applications occurs at the Layer 2 level, and VXLAN encapsulation or tunnels are used to transport their data packets over a Layer 3 network.

Figure 1: Using Overlay Ping and Traceroute to Troubleshoot a VXLAN Overlay NetworkUsing Overlay Ping and Traceroute to Troubleshoot a VXLAN Overlay Network

In this topology, there are two QFX5100 switches that function as VTEPs. VTEP1 initiates and terminates VXLAN tunnels for physical servers A and B, and VTEP2 does the same for physical server C. VTEP1 and VTEP2 are in VXLAN 100.

A data packet sent from physical server A is typically routed to the Layer 3 router with the IP address of 192.0.2.30 to reach physical server C.

In this VXLAN overlay network topology, a communication issue arises between physical servers A and C. To troubleshoot the issue with this data flow, you can initiate the ping overlay and traceroute overlay commands on VTEP1 (the source VTEP or tunnel-src) and specify that VTEP2 is the destination VTEP or tunnel-dst.

The ping overlay and traceroute overlay commands include several parameters. Table 1 explains the purpose and provides a value for each of the parameters used in scenarios 1, 2, and 3.

Table 1 does not include all available ping overlay and traceroute overlay parameters. This example uses the default values of these omitted parameters.

Table 1: Ping and Traceroute Overlay Parameter Values For Scenarios 1, 2, and 3

ping overlay and traceroute overlay Parameters

Description

Scenario to Which Parameter Applies

Value

tunnel-type

Identifies type of tunnel that you are troubleshooting.

All

vxlan

vni

VXLAN network identifier (VNI) of VXLAN used in this example.

All

100

tunnel-src

IP address of VTEP1, on which you initiate overlay ping or traceroute.

All

192.0.2.10

tunnel-dst

IP address of VTEP2, which receives the overlay ping or traceroute packets.

All

192.0.2.20

mac

MAC address of physical server C, which is the destination endpoint.

Scenarios 2 and 3 only

00:00:5E:00:53:cc

count

Number of overlay ping requests that VTEP1 sends.

Note:

The count parameter does not apply to overlay traceroute.

All

5

hash-source-mac

MAC address of physical server A, which is the source endpoint.

Scenario 3 only

00:00:5E:00:53:aa

hash-destination-mac

MAC address of physical server C, which is the destination endpoint.

Note:

When specifying this parameter for scenario 3, the MAC address must be the same MAC address as specified for the mac parameter.

Scenario 3 only

00:00:5E:00:53:cc

hash-source-address

IP address of physical server A.

Scenario 3 only

198.51.100.1

hash-destination-address

IP address of physical server C.

Scenario 3 only

198.51.100.3

hash-vlan

VLAN ID of source endpoint.

Note:

If the source endpoint is not a member of a VLAN, you do not need to use this parameter.

Scenario 3 only

150

hash-input-interface

VTEP1 interface on which data flow originates.

Scenario 3 only

xe-0/0/2

hash-protocol

A value for the protocol used in the data flow.

Scenario 3 only

17

hash-source-port

A value for the outer TCP/UDP source port.

Scenario 3 only

4456

hash-destination-port

A value for the outer UDP destination port.

Scenario 3 only

4540

Table 1 includes several hash parameters, which are used for scenario 3. For each of these parameters, you must specify a value associated with the data flow that you are troubleshooting. Based on the values that you specify, the system calculates a VXLAN UDP header source port hash, which is included in the VXLAN UDP header of the overlay ping and traceroute packets. Including the calculated hash in the VXLAN UDP header enables the overlay ping and traceroute packets to emulate data packets in the flow that you are troubleshooting.

Best Practice:

When using the hash parameters, we recommend that you specify a value for each parameter. The exception to this guideline is the hash-vlan parameter, which you do not have to use if the source endpoint is not a member of a VLAN. This practice ensures that the overlay ping and traceroute processes are successful and that the output for each command is accurate. If you do not specify a value for one or more of the hash parameters, the system sends an OAM request that might include incorrect hash values and generates a warning message.

Verification

This section includes the following verification tasks:

Scenario 1: Verifying That VXLAN 100 Is Configured on VTEP2

Purpose

Verify that a VXLAN with the VNI of 100 is configured on VTEP2. You can use either overlay ping or traceroute to perform this verification.

Action

Overlay Ping

On VTEP1, initiate an overlay ping:

Overlay Traceroute

On VTEP1, initiate an overlay traceroute:

Meaning

The sample overlay ping output indicates the following:

  • VTEP1 sent five ping requests to VTEP2, and VTEP2 responded to each request.

  • VTEP2 indicated that the VNI of 100 is not configured (Overlay-segment not present at RVTEP 192.0.2.20) and included this information in its response to VTEP1.

The sample overlay traceroute output indicates the following:

  • Upon receiving an overlay traceroute packet with a time-to-live (TTL) value of 1 hop, the Layer 3 router responds to VTEP1.

  • Upon receiving an overlay traceroute packet with a TTL value of 2 hops, VTEP2 responds to VTEP1.

  • VTEP2 indicated that the VNI of 100 is not configured (Overlay-segment not present at RVTEP 192.0.2.20) and included this information in its response to VTEP1.

Note:

The asterisk (*) in the Receiver Timestamp column of the overlay traceroute output indicates that the Layer 3 router that received the overlay traceroute packet is not a Juniper Networks device or is a Juniper Networks device that does not support overlay traceroute.

Given that the output of both overlay ping and traceroute indicates that VXLAN 100 is not present, check for this configuration on VTEP2. If you must configure a VNI of 100 on VTEP2, use the vni configuration statement at the [edit vlans vlan-id vxlan] hierarchy level, and reissue the ping overlay or traceroute overlay command to verify that VXLAN 100 is now recognized.

Scenario 2: Verifying That the MAC Address of the Destination Endpoint Is on VTEP2

Purpose

Verify that the MAC address (00:00:5E:00:53:cc) of physical server C, which is the destination endpoint, is in the forwarding table of VTEP2. You can use either overlay ping or traceroute to perform this verification.

Action

Overlay Ping

On VTEP1, initiate an overlay ping:

Overlay Traceroute

On VTEP1, initiate an overlay traceroute:

Meaning

The sample overlay ping output indicates the following:

  • VTEP1 sent five ping requests to VTEP2, and VTEP2 responded to each request.

  • VTEP2 verified that the VNI of 100 is configured (Overlay-segment present at RVTEP 192.0.2.20) but that the MAC address of physical server C is not in the forwarding table (End-System Not Present). VTEP2 included this information in its response to VTEP1.

The sample overlay traceroute output indicates the following:

  • Upon receiving an overlay traceroute packet with a TTL value of 1 hop, the Layer 3 router responds to VTEP1.

  • Upon receiving an overlay traceroute packet with a TTL value of 2 hops, VTEP2 responds to VTEP1.

  • VTEP2 verified that the VNI of 100 is configured (Overlay-segment present at RVTEP 192.0.2.20) but that the MAC address of physical server C is not in the forwarding table (End-System Not Present). VTEP2 included this information in its response to VTEP1.

Note:

The asterisk (*) in the Receiver Timestamp column of the overlay traceroute output indicates that the Layer 3 router that received the overlay traceroute packet is not a Juniper Networks device or is a Juniper Networks device that does not support overlay traceroute.

Given that the output of both overlay ping and traceroute indicates that the MAC address of physical server C is not known by VTEP2, you must further investigate to determine why this MAC address is not in the forwarding table of VTEP2.

Scenario 3: Verifying a Data Flow

Purpose

Verify that there are no issues that might impede the flow of data from physical server A to physical server C. The networking devices that support this flow include VTEP1, the Layer 3 router with the IP address of 192.0.2.30, and VTEP2 (see Figure 1).

Initially, use overlay ping, and if the overlay ping results indicate an issue, then use overlay traceroute to determine in which segment of the path the issue exists.

With both overlay ping and traceroute, use the hash parameters to specify information about the devices in this data flow so that the system can calculate a VXLAN UDP header source port hash, which is included in the VXLAN UDP header of the overlay ping and traceroute packets. With the calculated hash included in the VXLAN UDP header, the overlay ping and traceroute packets can emulate data packets in this flow, which should produce more accurate ping and traceroute results.

Best Practice:

When using the hash parameters, we recommend specifying a value for each parameter. The exception to this guideline is the hash-vlan parameter, which you do not have to use if the source endpoint is not a member of a VLAN. This practice ensures that the overlay ping and traceroute processes are successful and that the output for each command is accurate. If you do not specify a value for one or more of the hash parameters, the system sends an OAM request that might include incorrect hash values and generates a warning message.

Action

Overlay Ping

On VTEP1, initiate an overlay ping:

Overlay Traceroute

If needed, on VTEP1, initiate an overlay traceroute:

Meaning

The sample overlay ping output indicates that VTEP1 sent five ping requests to VTEP2, but VTEP2 did not respond to any of the requests. The lack of response from VTEP2 indicates that a connectivity issue exists along the path between VTEP1 and the Layer 3 router or the path between the Layer 3 router and VTEP2.

To further troubleshoot in which path the issue lies, overlay traceroute is used. The sample overlay traceroute output indicates the following:

  • Upon receiving an overlay traceroute packet with a TTL value of 1 hop, the Layer 3 router responds to VTEP1, which indicates that the path between VTEP1 and the Layer 3 router is up.

  • VTEP2 does not respond to the overlay traceroute packet, which indicates that the path between the Layer 3 router and VTEP2 might be down.

Note:

The asterisk (*) in the Receiver Timestamp column of the overlay traceroute output indicates that the Layer 3 router that received the overlay traceroute packet is not a Juniper Networks device or is a Juniper Networks device that does not support overlay traceroute.

Given that the overlay traceroute output indicates that there is a connectivity issue between the Layer 3 router and VTEP2, you must further investigate this path segment to determine the source of the issue.