Tracing the vRouter Packet Path
Contrail Networking vRouter is the component that takes packets from virtual machines (VM)s and forwards them to their destinations. Tracing is a useful tool for debugging the packet path.
In this topic, we trace the vRouter packet path in the following use cases:
Unicast Packet Path - Intra-VN
This procedure steps through debugging the unicast packet path
for intra-virtual network (intra-VN) traffic from VM1 to VM2 (on same
compute node) and VM3 (on different compute node). In this example,
the VMs listed are in the same subnet 10.1.1.0/24. Intra-VN traffic is within the same virtual network.
| VM1 | IP address |
| VM2 | IP address |
| VM3 | IP address |
Intra-Compute Use Case
Discover the
vifinterfaces corresponding to the virtual machine interfaces (VMI)s of the VM by using the command:vif --list
You can also discover the
vifinterfaces by entering the introspect URL.Example:
http://10.1.1.5:8085/Snh_ItfReq?name=&type=&uuid=&vn=&mac=&ipv4_address=&ipv6_address=&parent_uuid=&ip_active=&ip6_active=&l2_active=
Note:Replace the IP address with the actual compute IP address in the introspect HTTP URL.
Run the
vif --get <index>command to verify the virtual routing and forwarding (VRF) and Policy flags are set in the vRouter interface (VIF).Example output verifying flags for each
vif:vif0/4 OS: tapdd789d34-27 Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:10.1.1.5 Vrf:2 Mcast Vrf:2 Flags:PL3L2Er QOS:-1 Ref:6 RX packets:30 bytes:8676 errors:0 TX packets:170 bytes:7140 errors:0 ISID: 0 Bmac: 02:dd:78:9d:34:27 Drops:81 vif0/6 OS: tapaedbc037-bf Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:10.1.1.6 Vrf:2 Mcast Vrf:2 Flags:PL3L2Er QOS:-1 Ref:6 RX packets:96316 bytes:4858043 errors:0 TX packets:96562 bytes:4884177 errors:0 ISID: 0 Bmac: 02:ae:db:c0:37:bf Drops:2Run the following command to display all of the entries from the bridge table:
rt --dump <vrf id> --family bridge
Example:
rt --dump 2 --family bridge Flags: L=Label Valid, Df=DHCP flood, Mm=Mac Moved, L2c=L2 Evpn Control Word, N=New Entry, Ec=EvpnControlProcessing vRouter bridge table 0/2 Index DestMac Flags Label/VNID Nexthop Stats 31264 0:0:5e:0:1:0 Df - 3 206834 79784 2:dd:78:9d:34:27 Df - 44 4 112924 ff:ff:ff:ff:ff:ff LDf 5 48 35 115240 2:0:0:0:0:2 Df - 12 0 169408 2:ae:db:c0:37:bf Df - 25 1 183944 2:99:ef:64:96:e1 LDf 27 20 0 205564 2:0:0:0:0:1 Df - 12 0 252380 0:25:90:c5:58:94 Df - 3 0Highlighted in the example is the destination MAC address of the destination VM in the bridge table and the next-hop identifier associated with it.
Run
nh --get <nh id>to display the next-hop details.Example:
nh --get 25 Id:25 Type:Encap Fmly:AF_BRIDGE Rid:0 Ref_cnt:3 Vrf:2 Flags:Valid, Policy, Etree Root, EncapFmly:0806 Oif:6 Len:14 Encap Data: 02 ae db c0 37 bf 00 00 5e 00 01 00 08 00In the example,
Oif:6is the OIF index in the next hop which is the outgoing interface for the packet. TheEncap Datacorresponds to the L2 encapsulation that is added to the IP packet before the packet is forwarded to the outgoing interface.Run
vif --get <oifindex>to get the outgoing VIF details.Example:
vif --get 6 vif0/6 OS: tapaedbc037-bf Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:10.1.1.6 Vrf:2 Mcast Vrf:2 Flags:PL3L2Er QOS:-1 Ref:6 RX packets:96935 bytes:4892936 errors:0 TX packets:97235 bytes:4921004 errors:0 ISID: 0 Bmac: 02:ae:db:c0:37:bf Drops:2The received packet
RXand transmitted packetTXcounters for the corresponding VIF interfaces are incremented when traffic is flowing through.Run the
flow -lcommand to list the flows created. If the Policy flag is enabled on the VIFs, a flow is created as shown in the example.Example: Ping
10.1.1.6from10.1.1.5.flow -l Index Source:Port/Destination:Port Proto(V) ----------------------------------------------------------------------------------------- 876<=>1020 10.1.1.6:1599 1 (2) 10.1.1.5:0 (Gen: 12, K(nh):21, Action:F, Flags:, QOS:-1, S(nh):21, Stats:9/882, SPort 54920, TTL 0, Sinfo 6.0.0.0) 1020<=>876 10.1.1.5:1599 1 (2) 10.1.1.6:0 (Gen: 2, K(nh):29, Action:F, Flags:, QOS:-1, S(nh):29, Stats:9/882, SPort 58271, TTL 0, Sinfo 4.0.0.0)The statistics in the forward and reverse flow are incremented when traffic is flowing through. If statistics are not getting incremented for a particular flow, that can indicate a potential problem in that direction. The flow action should be
ForNfor the packets to be forwarded or NATed out. A flow action ofDindicates that packets will be dropped.Run the
vrouter_agent_debugscript to collect all of the relevant logs.
Inter-Compute Use Case
In an inter-compute case, the next-hop lookup points to the tunnel that takes the packet to the other compute node. The bridge entry will also indicate the Label/VNID added to the packet during encapsulation. Inter-compute traffic is between VMs on different compute nodes.
For Compute 1:
Discover the
vifinterfaces corresponding to the virtual machine interfaces (VMI)s of the VM by using the command:vif --list
You can also discover the
vifinterfaces by entering the introspect URL:Example:
http://10.1.1.5:8085/Snh_ItfReq?name=&type=&uuid=&vn=&mac=&ipv4_address=&ipv6_address=&parent_uuid=&ip_active=&ip6_active=&l2_active=
Note:Replace the IP address with the actual compute IP address in the introspect HTTP URL.
Run the
vif --get <index>command to verify the virtual routing and forwarding (VRF) and Policy flags are set in the vRouter interface (VIF).Example output verifying flags for each
vif:vif0/4 OS: tapdd789d34-27 Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:10.1.1.5 Vrf:2 Mcast Vrf:2 Flags:PL3L2Er QOS:-1 Ref:6 RX packets:30 bytes:8676 errors:0 TX packets:170 bytes:7140 errors:0 ISID: 0 Bmac: 02:dd:78:9d:34:27 Drops:81 vif0/6 OS: tapaedbc037-bf Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:10.1.1.6 Vrf:2 Mcast Vrf:2 Flags:PL3L2Er QOS:-1 Ref:6 RX packets:96316 bytes:4858043 errors:0 TX packets:96562 bytes:4884177 errors:0 ISID: 0 Bmac: 02:ae:db:c0:37:bfRun the following command to display all of the entries from the bridge table:
rt --dump <vrf id> --family bridge
Example:
rt --dump 2 --family bridge Flags: L=Label Valid, Df=DHCP flood, Mm=Mac Moved, L2c=L2 Evpn Control Word, N=New Entry, Ec=EvpnControlProcessing vRouter bridge table 0/2 Index DestMac Flags Label/VNID Nexthop Stats 31264 0:0:5e:0:1:0 Df - 3 206834 79784 2:dd:78:9d:34:27 Df - 44 4 112924 ff:ff:ff:ff:ff:ff LDf 5 48 35 115240 2:0:0:0:0:2 Df - 12 0 169408 2:ae:db:c0:37:bf Df - 25 1 183944 2:99:ef:64:96:e1 LDf 27 20 0 205564 2:0:0:0:0:1 Df - 12 0 252380 0:25:90:c5:58:94 Df - 3 0In the example,
2:99:ef:64:96:e1belongs to IP address10.1.1.7and label27is used to encapsulate the packet.Run
nh --get <nh id>to get the next hop details.Example:
nh --get 20 Id:20 Type:Tunnel Fmly: AF_INET Rid:0 Ref_cnt:12 Vrf:0 Flags:Valid, MPLSoGRE, Etree Root, Oif:0 Len:14 Data:00 25 90 c5 62 1c 00 25 90 c5 58 94 08 00 Sip:10.204.217.86 Dip:10.204.217.70
In the example, the next-hop output indicates the next-hop type as
Tunnel, encapsulation used asMPLSoGRE, the outgoing interface asOif:0, and the corresponding source and destination IP addresses of the tunnel.
For Compute 2:
Run the
mpls --get <label>command to see the next hop mapped to the particular incoming MPLS table.Example:
mpls --get 27 MPLS Input Label Map Label NextHop ------------------------- 27 28Run
nh --get <nh_id>to get the next hop details.Example:
nh --get 28 Id:28 Type:Encap Fmly:AF_BRIDGE Rid:0 Ref_cnt:3 Vrf:2 Flags:Valid, Policy, Etree Root, EncapFmly:0806 Oif:3 Len:14 Encap Data: 02 99 ef 64 96 e1 00 00 5e 00 01 00 08 00Run
vif --get <oifindex>to get the outgoing VIF details.Example:
vif --get 3 vif0/3 OS: tap99ef6496-e1 Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:10.1.1.7 Vrf:2 Mcast Vrf:2 Flags:PL3L2Er QOS:-1 Ref:6 RX packets:34 bytes:10044 errors:0 TX packets:1699 bytes:78990 errors:0 Drops:93Note:If you are using VXLAN encapsulation, do the following on Compute 2:
For Step 1, instead of running the
mpls --getcommand, run thevxlan --get <vxlanid>command to get the mapping from VXLAN ID to the next hop.With VXLAN, the next hop points to a VRF translated next hop. Use the bridge lookup in the corresponding VRF, as shown in Step 3 to get the final outgoing next hop, which will point to the VIF interface.
Unicast Packet Path - Inter-VN
The following procedure steps through debugging the packet path
from VM1 to VM2 (on the same compute node) and VM3 (on a different
compute node). In this example, the virtual machines (VMs) listed
are in the same subnet 10.1.1.0/24.
| VM1 | IP address |
| VM2 | IP address |
| VM3 | IP address |
Replace the IP address with the actual compute IP address in all of the introspect URLs.
Intra-Compute Use Case
Discover the
vifinterfaces corresponding to the virtual machine interfaces (VMI)s of the VM using the command:vif --list
You can also discover the
vifinterfaces by entering the introspect URL:Example:
http://10.1.1.5:8085/Snh_ItfReq?name=&type=&uuid=&vn=&mac=&ipv4_address=&ipv6_address=&parent_uuid=&ip_active=&ip6_active=&l2_active=
Note:Replace the IP address with the actual compute IP address in the introspect HTTP URLs.
Run the
vif --get <index>command to verify the virtual routing and forwarding (VRF) and Policy flags are set in the vRouter interface (VIF).Example output verifying flags for each
vif:vif0/4 OS: tapdd789d34-27 Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:10.1.1.5 Vrf:2 Mcast Vrf:2 Flags:PL3L2Er QOS:-1 Ref:6 RX packets:30 bytes:8676 errors:0 TX packets:170 bytes:7140 errors:0 ISID: 0 Bmac: 02:dd:78:9d:34:27 Drops:81 vif0/6 OS: tapaedbc037-bf Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:10.1.1.6 Vrf:2 Mcast Vrf:2 Flags:PL3L2Er QOS:-1 Ref:6 RX packets:96316 bytes:4858043 errors:0 TX packets:96562 bytes:4884177 errors:0 ISID: 0 Bmac: 02:ae:db:c0:37:bf Drops:2Run the following command to display all of the entries from the bridge table:
rt --dump <vrf id> --family bridge
Example:
rt --dump 2 --family bridge Flags: L=Label Valid, Df=DHCP flood, Mm=Mac Moved, L2c=L2 Evpn Control Word, N=New Entry, Ec=EvpnControlProcessing vRouter bridge table 0/2 Index DestMac Flags Label/VNID Nexthop Stats 31264 0:0:5e:0:1:0 Df - 3 212744 79784 2:dd:78:9d:34:27 Df - 44 408 112924 ff:ff:ff:ff:ff:ff LDf 5 51 38 115240 2:0:0:0:0:2 Df - 12 0 169408 2:ae:db:c0:37:bf Df - 25 405 183944 2:99:ef:64:96:e1 LDf 5 37 0 205564 2:0:0:0:0:1 Df - 12 0 252380 0:25:90:c5:58:94 Df - 3 0In the case of inter-virtual network (VN)s, the packets are Layer 3 routed instead of Layer 2 switched. The vRouter does a proxy ARP for the destination network providing it’s virtual MAC address
0:0:5e:0:1:0back for the ARP request from the source. This can be seen from thert –dumpof the source VN inet table. This results in the packet being received by the vRouter, which does the route lookup to send the packet to the correct destination.Run
nh --get <nh id>to display the next-hop details.Example:
nh --get 3 Id:3 Type:L2 Receive Fmly: AF_INET Rid:0 Ref_cnt:8 Vrf:0 Flags:Valid, Etree Root,Run
rt --dump 2 --family inet | grep <ip address>to display inet family routes on the specified IP address.Example:
rt --dump 2 --family inet | grep 20.1.1.6 Destination PPL Flags Label Nexthop Stitched MAC(Index) 20.1.1.6/32 32 P - 30 - 20.1.1.60/32 0 - 0 - 20.1.1.61/32 0 - 0 - 20.1.1.62/32 0 - 0 - 20.1.1.63/32 0 - 0 - 20.1.1.64/32 0 - 0 - 20.1.1.65/32 0 - 0 - 20.1.1.66/32 0 - 0 - 20.1.1.67/32 0 - 0 - 20.1.1.68/32 0 - 0 - 20.1.1.69/32 0 - 0 - 20.1.106.0/24 0 - 0 -
Run
nh --get <nh id>to get the next hop details.Example:
nh --get 30 Id:30 Type:Encap Fmly: AF_INET Rid:0 Ref_cnt:5 Vrf:3 Flags:Valid, Policy, Etree Root, EncapFmly:0806 Oif:3 Len:14 Encap Data: 02 60 fc 55 cb bf 00 00 5e 00 01 00 08 00Run
vif --get <oifindex>to get the outgoing VIF details.Example:
vif --get 3 vif0/3 OS: tap60fc55cb-bf Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:20.1.1.6 Vrf:3 Mcast Vrf:3 Flags:PL3L2Er QOS:-1 Ref:6 RX packets:356 bytes:32586 errors:0 TX packets:3930 bytes:177290 errors:0 ISID: 0 Bmac: 02:60:fc:55:cb:bf Drops:84
Inter-Compute Use Case
In the case of inter-compute, the next hop looked up to send the packet out, will point to a tunnel next hop. Depending on the encapsulation priority, the appropriate encapsulation is added and the packet is tunneled out. Inter-compute traffic is between VMs on different compute nodes.
For Compute 1:
Run
rt --dump 2 --family inet | grep <ip address>to display inet family routes for a specified IP address.Example:
rt --dump 2 --family inet | grep 20.1.1.5 20.1.1.5/32 32 LP 32 49 - 20.1.1.50/32 0 - 0 - 20.1.1.51/32 0 - 0 - 20.1.1.52/32 0 - 0 - 20.1.1.53/32 0 - 0 - 20.1.1.54/32 0 - 0 - 20.1.1.55/32 0 - 0 - 20.1.1.56/32 0 - 0 - 20.1.1.57/32 0 - 0 - 20.1.1.58/32 0 - 0 -
Run
nh --get <nh id>to display the next-hop details, which points to a tunnel next hop.Example:
nh --get 49 Id:49 Type:Tunnel Fmly: AF_INET Rid:0 Ref_cnt:9 Vrf:0 Flags:Valid, MPLSoUDP, Etree Root, Oif:0 Len:14 Data:00 25 90 c5 62 1c 00 25 90 c5 58 94 08 00 Sip:10.204.217.86 Dip:10.204.217.70
For Compute 2:
Run the
mpls --get <label>command to see the next hop mapped to the particular incoming MPLS table.Example:
mpls --get 32 MPLS Input Label Map Label NextHop ------------------------- 32 36Run
nh --get <nh id>to view the next hop details.Example:
nh --get 36 Id:36 Type:Encap Fmly: AF_INET Rid:0 Ref_cnt:5 Vrf:3 Flags:Valid, Policy, Etree Root, EncapFmly:0806 Oif:4 Len:14 Encap Data: 02 f3 37 7b 53 25 00 00 5e 00 01 00 08 00In the example,
Oif:4is the OIF index in the next hop which is the outgoing interface for the packet. TheEncap Datacorresponds to the L2 encapsulation that is added to the IP packet before the packet is forwarded to the outgoing interface.Run
vif --get <oifindex>to get the outgoing VIF details.Example:
vif --get 4 vif0/4 OS: tapf3377b53-25 Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:20.1.1.5 Vrf:3 Mcast Vrf:3 Flags:PL3L2Er QOS:-1 Ref:6 RX packets:100 bytes:16915 errors:0 TX packets:3955 bytes:178363 errors:0 ISID: 0 Bmac: 02:f3:37:7b:53:25 Drops:78For details about EVPN type 5 routing in Contrail Networking, see Support for EVPN Route Type 5.
Broadcast, Unknown Unicast, and Multicast Packet Path
The following procedure steps through debugging the packet path for broadcast, unknown unicast, and multicast (BUM) traffic in Contrail Networking. In this example, the virtual machines (VMs) listed are in the same subnet 70.70.70.0/24.
The ToR Service Node (TSN) actively holds the contrail-tor-agent and is responsible for:
Acting as a receiver of all BUM traffic coming from the ToR switch.
Acting as DNS/DHCP responder for the BMS connected to the ToR switch.
Contrail Networking releases earlier than 5.x, used an Open vSwitch Database (OVSDB)-managed VXLAN environment.
Topology example for an OVSDB-managed VXLAN:
Top-of-Rack Switch 1 (ToR SW1) - 10.204.74.229 (lo0.0 = 1.1.1.229)
Top-of-Rack Switch 2 (ToR SW2) - 10.204.74.230 (lo0.0 = 1.1.1.230)
ToR Services Node 1 (TSN1) = 10.219.94.7
ToR Services Node 2 (TSN2) = 10.219.94.8
Controller1 = 10.219.94.4
Controller2 = 10.219.94.5
Controller3 = 10.219.94.6
Compute1 = 10.219.94.9
Compute2 = 19.219.94.18
Virtual Network (VN) = 70.70.70.0/24
Virtual Machine 1 (VM1) = 70.70.70.3 residing on Compute2
Virtual Machine 2 (VM2) = 70.70.70.5 residing on Compute1
Bare Metal Server 1 (BMS1) = 70.70.70.100
Bare Metal Server 2 (BMS2) = 70.70.70.101
Run the
set protocols ovsdb interfaces <interface>command to configure the physical interfaces that you want the OVSDB protocol to manage.Example:
set protocols ovsdb interfaces ge-0/0/46 set protocols ovsdb interfaces ge-0/0/90
The ToR interfaces from which the BMS hangs are marked as
ovsdb interfaces.View packets coming into these interfaces by displaying the
ovsdb mactable for the ToR switch.Example:
root@QFX5100-229> show ovsdb mac Logical Switch Name: Contrail-9ed1f70a-6eac-4cdb-837a-1579728fd9a1 Mac IP Encapsulation Vtep Address Address Address ff:ff:ff:ff:ff:ff 0.0.0.0 Vxlan over Ipv4 1.1.1.229 40:a6:77:d8:37:1d 0.0.0.0 Vxlan over Ipv4 1.1.1.229 02:3b:ce:56:61:98 0.0.0.0 Vxlan over Ipv4 10.219.94.18 02:53:89:c4:29:1c 0.0.0.0 Vxlan over Ipv4 10.219.94.18 02:72:e9:7a:cd:f5 0.0.0.0 Vxlan over Ipv4 10.219.94.9 02:75:a1:33:65:3c 0.0.0.0 Vxlan over Ipv4 10.219.94.9 ff:ff:ff:ff:ff:ff 0.0.0.0 Vxlan over Ipv4 10.219.94.7 {master:0}
The entry marked in red (
ff:ff:ff:ff:ff:ff:ff- broadcast route) indicates the next hop for a BUM packet coming into the ToR SW’sovsdb interface. In this case, VTEP address10.219.94.7is the next hop, which is TSN1. This changes based on which TSN has the activecontrail-tor-agentfor the ToR switch in question. With this, the BUM packet is forwarded to the TSN node in a VXLAN tunnel (local VTEP source interface is1.1.1.229and RVTEP source interface is10.219.94.7).The VXLAN encapsulated packet is sent with a VXLAN Network Identifier (VNI) that is predetermined by Contrail Networking when logical interfaces are created. For example, when
ge-0/0/46was configured as a logical port in Contrail Networking, the following configuration was committed on the ToR.Example:
set vlans Contrail-9ed1f70a-6eac-4cdb-837a-1579728fd9a1 interface ge-0/0/46.0 set vlans Contrail-9ed1f70a-6eac-4cdb-837a-1579728fd9a1 interface ge-0/0/90.0 set vlans Contrail-9ed1f70a-6eac-4cdb-837a-1579728fd9a1 vxlan vni 4As the VXLAN encapsulated packet arrives on the TSN node, let’s examine how the vRouter handles this packet.
Run
vxlan --dumpto dump the VXLAN table. The VXLAN table maps a network ID to a next hop.Example:
root@contrail61:~# vxlan --dump VXLAN Table VNID NextHop ---------------- 4 13In the example, next hop
13is programmed for VNI4.Run
nh --get <nh id>to display the next-hop details and determine the virtual routing and forwarding (VRF) associated.Example:
root@contrail61:~# nh --get 13 Id:13 Type:Vrf_Translate Fmly: AF_INET Rid:0 Ref_cnt:2 Vrf:1 Flags:Valid, Vxlan, Vrf:1Run the following command to display all of the entries from the bridge table:
rt --dump <vrf id> --family bridge
Example:
root@contrail61:~# rt --dump 1 --family bridge Flags: L=Label Valid, Df=DHCP flood vRouter bridge table 0/1 Index DestMac Flags Label/VNID Nexthop 30780 0:1a:a0:e:30:26 - 1 57812 2:53:89:c4:29:1c LDf 17 15 70532 2:3b:ce:56:61:98 LDf 20 15 87024 2:72:e9:7a:cd:f5 LDf 17 14 97192 ff:ff:ff:ff:ff:ff LDf 4 24 121160 0:1a:a0:a:b4:87 - 1 225832 40:a6:77:d8:37:1d LDf 4 19 237084 0:11:a:6c:50:d4 Df - 3 244992 aa:bb:cc:dd:3e:f4 - 1 252916 0:0:5e:0:1:0 Df - 3 256476 2:75:a1:33:65:3c LDf 20 14
In the example bridge table, since we are tracing the BUM packet path, we need to examine the ff:ff:ff:ff:ff:ff:ff route by selecting the next hop programmed. In the example, it is 24. Note that a series of composite next hops are programmed.
Run
nh --get <nh id>to display the next-hop details.Example:
root@contrail61:~# nh --get 24 Id:24 Type:Composite Fmly:AF_BRIDGE Rid:0 Ref_cnt:4 Vrf:1 Flags:Valid, Multicast, L2, Sub NH(label): 20(0) 22(0) 21(0) Id:20 Type:Composite Fmly: AF_INET Rid:0 Ref_cnt:2 Vrf:1 Flags:Valid, Tor, Sub NH(label): 19(4) Id:19 Type:Tunnel Fmly: AF_INET Rid:0 Ref_cnt:3 Vrf:0 Flags:Valid, Vxlan, Oif:0 Len:14 Flags Valid, Vxlan, Data:00 00 5e 00 01 21 00 11 0a 6c 50 d4 08 00 Vrf:0 Sip:10.219.94.7 Dip:1.1.1.229 << Source where the BUM Traffic came from. Id:22 Type:Composite Fmly: AF_INET Rid:0 Ref_cnt:2 Vrf:1 Flags:Valid, Evpn, Sub NH(label): Id:21 Type:Composite Fmly: AF_INET Rid:0 Ref_cnt:2 Vrf:1 Flags:Valid, Fabric, Sub NH(label): 15(4099) Id:15 Type:Tunnel Fmly: AF_INET Rid:0 Ref_cnt:6 Vrf:0 Flags:Valid, MPLSoGRE, Oif:0 Len:14 Flags Valid, MPLSoGRE, Data:f8 bc 12 33 43 31 00 11 0a 6c 50 d4 08 00 Vrf:0 Sip:10.219.94.7 Dip:10.219.94.18 << Compute node which has a VM in this VN.The multicast tree in the example shows that there are two Dynamic IPs (DIP)s. The DIP where the packet came from is ignored. Therefore, packet gets forwarded to DIP
10.219.94.18only.Run
vxlan --get <vnid>to examine what DIP10.219.94.18does with the incoming VXLAN encapsulated packet.Example:
root@contrail4:~# vxlan --get 4 VXLAN Table VNID NextHop ---------------- 4 20Run
nh --get <nh id>to display the next-hop details.Example:
root@contrail4:~# nh --get 20 Id:20 Type:Vrf_Translate Fmly: AF_INET Rid:0 Ref_cnt:2 Vrf:1 Flags:Valid, Vxlan, Vrf:1Run the following command to display all of the entries from the bridge table:
rt --dump <vrf id> --family bridge
Example:
root@contrail4:~# rt --dump 1 --family bridge Flags: L=Label Valid, Df=DHCP flood vRouter bridge table 0/1 Index DestMac Flags Label/VNID Nexthop 57812 2:53:89:c4:29:1c - 15 70532 2:3b:ce:56:61:98 - 22 87024 2:72:e9:7a:cd:f5 LDf 17 25 97192 ff:ff:ff:ff:ff:ff LDf 4 50 112856 f8:bc:12:33:43:31 Df - 3 225832 40:a6:77:d8:37:1d LDf 4 18 252916 0:0:5e:0:1:0 Df - 3 256476 2:75:a1:33:65:3c LDf 20 25
In the example bridge table, since we are tracing the BUM packet path, we need to examine the ff:ff:ff:ff:ff:ff:ff route by selecting the next hop programmed. In the example, it is 50.
Run
nh --get <nh id>to display the next-hop details.Example:
root@contrail4:~# nh --get 50 Id:50 Type:Composite Fmly:AF_BRIDGE Rid:0 Ref_cnt:4 Vrf:1 Flags:Valid, Multicast, L2, Sub NH(label): 43(0) 49(0) Id:43 Type:Composite Fmly: AF_INET Rid:0 Ref_cnt:2 Vrf:1 Flags:Valid, Fabric, Sub NH(label): 31(4612) 25(4617) Id:31 Type:Tunnel Fmly: AF_INET Rid:0 Ref_cnt:2 Vrf:0 Flags:Valid, MPLSoGRE, Oif:0 Len:14 Flags Valid, MPLSoGRE, Data:00 11 0a 6c 50 d4 f8 bc 12 33 43 31 08 00 Vrf:0 Sip:10.219.94.18 Dip:10.219.94.7 <<< Source where the BUM traffic came from. Id:25 Type:Tunnel Fmly: AF_INET Rid:0 Ref_cnt:2562 Vrf:0 Flags:Valid, MPLSoGRE, Oif:0 Len:14 Flags Valid, MPLSoGRE, Data:44 a8 42 3a 94 f4 f8 bc 12 33 43 31 08 00 Vrf:0 Sip:10.219.94.18 Dip:10.219.94.9 <<< Compute node which has a VM in this VN. Id:49 Type:Composite Fmly: AF_INET Rid:0 Ref_cnt:2 Vrf:1 Flags:Valid, Encap, Sub NH(label): 14(0) 21(0) Id:14 Type:Encap Fmly:AF_BRIDGE Rid:0 Ref_cnt:2 Vrf:1 Flags:Valid, EncapFmly:0806 Oif:4 Len:14 Encap Data: 02 53 89 c4 29 1c 00 00 5e 00 01 00 08 00 Id:21 Type:Encap Fmly:AF_BRIDGE Rid:0 Ref_cnt:2 Vrf:1 Flags:Valid, EncapFmly:0806 Oif:3 Len:14 Encap Data: 02 3b ce 56 61 98 00 00 5e 00 01 00 08 00 <<< Local VM belonging to this VN that is an intended receiver of this multicast traffic.In the example, you only have to inspect DIP
10.219.94.9. The remaining endpoints are either local or the source where the BUM traffic came from. Now, let us examine, what DIP10.219.94.9does with the incoming VXLAN encapsulated packet.Run
vxlan --get <vnid>to examine what DIP10.219.94.9does with the incoming VXLAN encapsulated packet.Example:
root@contrail101:~# vxlan --get 4 VXLAN Table VNID NextHop ---------------- 4 20Run
nh --get <nh id>to display the next-hop details.Example:
root@contrail101:~# nh --get 20 Id:20 Type:Vrf_Translate Fmly: AF_INET Rid:0 Ref_cnt:2 Vrf:1 Flags:Valid, Vxlan, Vrf:1Display the bridge table for the VRF by using the following command:
rt --dump <vrf id> --family bridge
Example:
root@contrail101:~# rt --dump 1 --family bridge Flags: L=Label Valid, Df=DHCP flood vRouter bridge table 0/1 Index DestMac Flags Label/VNID Nexthop 57812 2:53:89:c4:29:1c LDf 17 28 70532 2:3b:ce:56:61:98 LDf 20 28 87024 2:72:e9:7a:cd:f5 - 15 97192 ff:ff:ff:ff:ff:ff LDf 4 31 140744 44:a8:42:3a:94:f4 Df - 3 225832 40:a6:77:d8:37:1d LDf 4 24 252916 0:0:5e:0:1:0 Df - 3 256476 2:75:a1:33:65:3c - 22 Encap Data: f8 bc 12 33 43 31 44 a8 42 3a 94 f4 08 00
Run
nh --get <nh id>to display the next-hop details.Example:
root@contrail101:~# nh --get 31 Id:31 Type:Composite Fmly:AF_BRIDGE Rid:0 Ref_cnt:4 Vrf:1 Flags:Valid, Multicast, L2, Sub NH(label): 30(0) 36(0) Id:30 Type:Composite Fmly: AF_INET Rid:0 Ref_cnt:2 Vrf:1 Flags:Valid, Fabric, Sub NH(label): 29(4612) 28(4099) Id:29 Type:Tunnel Fmly: AF_INET Rid:0 Ref_cnt:2 Vrf:0 Flags:Valid, MPLSoGRE, Oif:0 Len:14 Flags Valid, MPLSoGRE, Data:00 11 0a 6c 50 ac 44 a8 42 3a 94 f4 08 00 Vrf:0 Sip:10.219.94.9 Dip:10.219.94.8 << TSN2 in this topology that is managing a ToR with an end-point belonging to this VN. Id:28 Type:Tunnel Fmly: AF_INET Rid:0 Ref_cnt:2566 Vrf:0 Flags:Valid, MPLSoGRE, Oif:0 Len:14 Flags Valid, MPLSoGRE, Data:f8 bc 12 33 43 31 44 a8 42 3a 94 f4 08 00 Vrf:0 Sip:10.219.94.9 Dip:10.219.94.18 << Source where the BUM traffic came from. Id:36 Type:Composite Fmly: AF_INET Rid:0 Ref_cnt:2 Vrf:1 Flags:Valid, Encap, Sub NH(label): 14(0) 21(0) Id:14 Type:Encap Fmly:AF_BRIDGE Rid:0 Ref_cnt:2 Vrf:1 Flags:Valid, EncapFmly:0806 Oif:3 Len:14 Encap Data: 02 72 e9 7a cd f5 00 00 5e 00 01 00 08 00 << local VM that is an intended receiver of this traffic as it is tagged to this VN Id:21 Type:Encap Fmly:AF_BRIDGE Rid:0 Ref_cnt:2 Vrf:1 Flags:Valid, EncapFmly:0806 Oif:4 Len:14 Encap Data: 02 75 a1 33 65 3c 00 00 5e 00 01 00 08 00 << Local VM that is an intended receiver of this traffic since it is tagged to this VN.From the above output, the only DIP that you have to further examine is
10.219.94.8. The remaining DIPs are either local or the source where the BUM traffic came from. Now, let’s examine what DIP10.219.94.8does with the incoming VXLAN encapsulated packet.Run
vxlan --get <vnid>to examine what DIP10.219.94.9does with the incoming VXLAN encapsulated packet.Example:
root@contrail66:~# vxlan --get 4 VXLAN Table VNID NextHop ---------------- 4 14Run
nh --get <nh id>to display the next-hop details.Example:
root@contrail66:~# nh --get 14 Id:14 Type:Vrf_Translate Fmly: AF_INET Rid:0 Ref_cnt:2 Vrf:1 Flags:Valid, Vxlan, Vrf:1Display the bridge table for the VRF by using the following command:
rt --dump <vrf id> --family bridge
Example:
root@contrail66:~# rt --dump 1 --family bridge Flags: L=Label Valid, Df=DHCP flood vRouter bridge table 0/1 Index DestMac Flags Label/VNID Nexthop 30780 0:1a:a0:e:30:26 - 1 57812 2:53:89:c4:29:1c LDf 17 17 70532 2:3b:ce:56:61:98 LDf 20 17 87024 2:72:e9:7a:cd:f5 LDf 17 16 97192 ff:ff:ff:ff:ff:ff LDf 4 24 121160 0:1a:a0:a:b4:87 - 1 217208 0:11:a:6c:50:ac Df - 3 225832 40:a6:77:d8:37:1d LDf 4 20 244992 aa:bb:cc:dd:3e:f4 - 1 252916 0:0:5e:0:1:0 Df - 3 256476 2:75:a1:33:65:3c LDf 20 16
Run
nh --get <nh id>to display the next-hop details.Example:
root@contrail66:~# nh --get 24 Id:24 Type:Composite Fmly:AF_BRIDGE Rid:0 Ref_cnt:4 Vrf:1 Flags:Valid, Multicast, L2, Sub NH(label): 23(0) 25(0) 21(0) Id:23 Type:Composite Fmly: AF_INET Rid:0 Ref_cnt:2 Vrf:1 Flags:Valid, Tor, Sub NH(label): 22(4) Id:22 Type:Tunnel Fmly: AF_INET Rid:0 Ref_cnt:2 Vrf:0 Flags:Valid, Vxlan, Oif:0 Len:14 Flags Valid, Vxlan, Data:00 00 5e 00 01 21 00 11 0a 6c 50 ac 08 00 Vrf:0 Sip:10.219.94.8 Dip:1.1.1.230 <<< Another ToR switch that has an end-point belonging to this VN. Id:25 Type:Composite Fmly: AF_INET Rid:0 Ref_cnt:2 Vrf:1 Flags:Valid, Evpn, Sub NH(label): Id:21 Type:Composite Fmly: AF_INET Rid:0 Ref_cnt:2 Vrf:1 Flags:Valid, Fabric, Sub NH(label): 16(4617) Id:16 Type:Tunnel Fmly: AF_INET Rid:0 Ref_cnt:6 Vrf:0 Flags:Valid, MPLSoGRE, Oif:0 Len:14 Flags Valid, MPLSoGRE, Data:44 a8 42 3a 94 f4 00 11 0a 6c 50 ac 08 00 Vrf:0 Sip:10.219.94.8 Dip:10.219.94.9 <<< Source where the BUM traffic came from.Now, you just have one DIP
1.1.1.230which is the ToR SW2 in the topology. This should also be present in the multicast tree as this ToR SW also has an end-point (which is BMS2) in the same VN (VNI=4) as the one we are tracing.
This completes all levels of forwarding and tracing the BUM packet from one ToR switch and is replicated to other intended receivers in the topology.
These multicast trees are programmed by the controllers that the TSN is connected to. If you want to inspect the controller’s memory and what eventually gets programmed on all TSN computes, enter the following introspect URL using your controller IP address:
http://<controller_ip>:8083/Snh_ShowMulticastManagerDetailReq?x=default-domain:admin:seventy-network:seventy-network.ermvpn.0