Debugging EVPN VXLAN on RPD and Linux
SUMMARY
Before you start debugging for EVPN over VXLAN support in cRPD, ensure you have the configuration created.
Configuring EVPN Over VXLAN
routing-instances { evpn-vxlan { instance-type mac-vrf; protocols { evpn { encapsulation vxlan; default-gateway do-not-advertise; } } service-type vlan-aware; vtep-source-interface lo.0; bridge-domains { bd600 { vlan-id 600; interface ens3f2.600; routing-interface irb.600; vxlan { vni 2600; destination-udp-port 4790; } } bd601 { vlan-id 601; interface ens3f3.601; routing-interface irb.601; vxlan { vni 2601; destination-udp-port 4790; } } } route-distinguisher 81.1.1.1:1; vrf-target target:1:1; } } interfaces { irb { unit 600 { family inet { address 99.60.0.254/24; } family inet6 { address 1234::99.60.0.254/120; } } unit 601 { family inet { address 99.60.1.254/24; } family inet6 { address 1234::99.60.1.254/120; } } } }
Verifying Layer 2 EVPN Over VXLAN Support in cRPD
-
Verify the bridge device is created in RPD and Linux kernel.
root@PE1_CRPD> show evpn instance evpn-vxlan extensive
RPD view
Instance: evpn-vxlan Route Distinguisher: 81.1.1.1:1 Encapsulation type: VXLAN Control word enabled Duplicate MAC detection threshold: 5 Duplicate MAC detection window: 180 MAC database status Local Remote MAC advertisements: 3 2 MAC+IP advertisements: 9 6 Default gateway MAC advertisements: 2 0 Number of local interfaces: 3 (3 up) Interface name ESI Mode Status AC-Role .local..2 00:00:00:00:00:00:00:00:00:00 single-homed Up Root ens3f2.600 00:00:00:00:00:00:00:00:00:00 single-homed Up Root ens3f3.601 00:00:00:00:00:00:00:00:00:00 single-homed Up Root Number of IRB interfaces: 2 (2 up) Interface name VLAN VNI Status L3 context irb.600 2600 Up evpn-vrf irb.601 2601 Up evpn-vrf Number of protect interfaces: 0 Number of bridge domains: 2 VLAN Domain-ID Intfs/up IRB-intf Mode MAC-sync IM-label MAC-label v4-SG-sync IM-core-NH v6-SG-sync IM-core-NH Trans-ID 600 2600 1 1 irb.600 Extended Enabled 2600 Disabled Disabled 2600 601 2601 1 1 irb.601 Extended Enabled 2601 Disabled Disabled 2601 Number of neighbors: 1 Address MAC MAC+IP AD IM ES Leaf-label Remote-DCI-Peer 81.2.2.2 2 6 0 2 0 Number of ethernet segments: 2 ESI: 05:00:00:00:7b:00:00:0a:28:00 Local interface: irb.600, Status: Up/Forwarding ESI: 05:00:00:00:7b:00:00:0a:29:00 Local interface: irb.601, Status: Up/Forwarding Router-ID: 81.1.1.1 Source VTEP interface IP: 81.1.1.1 SMET Forwarding: Disabled
root@PE1_CRPD> show krt table | grep evpn-vxlan
evpn-vxlan.evpn-mac.0 : GF: 11 krt-index: 7 ID: 0 kernel-id: 2
Kernel view
root@PE1_CRPD:/# ip link show __crpd-brd2
__crpd-brd<2> is kernel id from show krt table
148: __crpd-brd2: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 56:68:a3:1a:07:9c brd ff:ff:ff:ff:ff:ff alias evpn-vxlan
root@PE1_CRPD:/# ip -d link show __crpd-brd2
148: __crpd-brd2: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 56:68:a3:1a:07:9c brd ff:ff:ff:ff:ff:ff promiscuity 0 bridge forward_delay 1500 hello_time 200 max_age 2000 ageing_time 30000 stp_state 0 priority 32768 vlan_filtering 1 vlan_protocol 802.1Q bridge_id 8000.56:68:a3:1a:7:9c designated_root 8000.56:68:a3:1a:7:9c root_port 0 root_path_cost 0 topology_change 0 topology_change_detected 0 hello_timer 0.00 tcn_timer 0.00 topology_change_timer 0.00 gc_timer 54.32 vlan_default_pvid 0 vlan_stats_enabled 0 group_fwd_mask 0 group_address 01:80:c2:00:00:00 mcast_snooping 0 mcast_router 1 mcast_query_use_ifaddr 0 mcast_querier 0 mcast_hash_elasticity 4 mcast_hash_max 512 mcast_last_member_count 2 mcast_startup_query_count 2 mcast_last_member_interval 100 mcast_membership_interval 26000 mcast_querier_interval 25500 mcast_query_interval 12500 mcast_query_response_interval 1000 mcast_startup_query_interval 3124 mcast_stats_enabled 0 mcast_igmp_version 2 mcast_mld_version 1 nf_call_iptables 0 nf_call_ip6tables 0 nf_call_arptables 0 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 alias evpn-vxlan
-
Verify if the VXLAN devices are created corresponding to the VXLAN
configuration under bridge domains.
RPD view
VXLAN configs of interest under routing-instance bridge-domains.
routing-instances { evpn-vxlan { bridge-domains { bd600 { ... vxlan { vni 2600; destination-udp-port 4790; } } bd601 { ... vxlan { vni 2601; destination-udp-port 4790; } } } } }
Kernel view
root@PE1_CRPD:/# ip -d link show vxlan2600
16: vxlan2600: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master __crpd-brd2 state UNKNOWN mode DEFAULT group default qlen 1000 link/ether 0e:6b:fd:27:a5:63 brd ff:ff:ff:ff:ff:ff promiscuity 1 vxlan id 2600 local 81.1.1.1 srcport 0 0 dstport 4790 nolearning tos inherit ttl 100 ageing 300 noudpcsum noudp6zerocsumtx noudp6zerocsumrx bridge_slave state forwarding priority 32 cost 100 hairpin off guard off root_block off fastleave off learning off flood on port_id 0x8003 port_no 0x3 designated_port 32771 designated_cost 0 designated_bridge 8000.e:6b:fd:27:a5:63 designated_root 8000.e:6b:fd:27:a5:63 hold_timer 0.00 message_age_timer 0.00 forward_delay_timer 0.00 topology_change_ack 0 config_pending 0 proxy_arp off proxy_arp_wifi off mcast_router 1 mcast_fast_leave off mcast_flood on neigh_suppress on group_fwd_mask 0x0 group_fwd_mask_str 0x0 vlan_tunnel off addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
root@PE1_CRPD:/# ip -d link show vxlan2601
17: vxlan2601: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master __crpd-brd2 state UNKNOWN mode DEFAULT group default qlen 1000 link/ether 32:82:1d:c2:e9:8b brd ff:ff:ff:ff:ff:ff promiscuity 1 vxlan id 2601 local 81.1.1.1 srcportdstport 4790 0 0 nolearning tos inherit ttl 100 ageing 300 noudpcsum noudp6zerocsumtx noudp6zerocsumrx bridge_slave state forwarding priority 32 cost 100 hairpin off guard off root_block off fastleave off learning off flood on port_id 0x8004 port_no 0x4 designated_port 32772 designated_cost 0 designated_bridge 8000.e:6b:fd:27:a5:63 designated_root 8000.e:6b:fd:27:a5:63 hold_timer 0.00 message_age_timer 0.00 forward_delay_timer 0.00 topology_change_ack 0 config_pending 0 proxy_arp off proxy_arp_wifi off mcast_router 1 mcast_fast_leave off mcast_flood on neigh_suppress on group_fwd_mask 0x0 group_fwd_mask_str 0x0 vlan_tunnel off addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
-
Verify all the instance interfaces (bridge domain interfaces including
vxlan devices) are enslaved to bridge device in kernel.
RPD view
Interface configs of interest under routing-instance bridge-domains.
routing-instances { evpn-vxlan { ... bridge-domains { bd600 { ... interface ens3f2.600; vxlan { vni 2600; -> vxlan2600 } } bd601 { ... interface ens3f3.601; vxlan { vni 2601; -> vxlan2601 } } } } }
Kernel view
Ensure all the instance IFL have "master __crpd-brd2" which means they are ensalved to __crpd-brd2 bridge device through ip link.
root@PE1_CRPD:/# ip link show master __crpd-brd2
12: ens3f2.600@ens3f2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master __crpd-brd2 state UP mode DEFAULT group default qlen 1000 link/ether 56:68:a3:54:20:b7 brd ff:ff:ff:ff:ff:ff 13: ens3f3.601@ens3f3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master __crpd-brd2 state UP mode DEFAULT group default qlen 1000 link/ether 56:68:a3:54:20:bb brd ff:ff:ff:ff:ff:ff 16: vxlan2600: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master __crpd-brd2 state UNKNOWN mode DEFAULT group default qlen 1000 link/ether 0e:6b:fd:27:a5:63 brd ff:ff:ff:ff:ff:ff 17: vxlan2601: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master __crpd-brd2 state UNKNOWN mode DEFAULT group default qlen 1000 link/ether 32:82:1d:c2:e9:8b brd ff:ff:ff:ff:ff:ff 19: irbbe-brd2@irbve-brd2: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master __crpd-brd2 state UP mode DEFAULT group default qlen 1000 link/ether fe:72:e9:b0:b5:92 brd ff:ff:ff:ff:ff:ff
-
Verify if all the instance interfaces which are part of the bridge device
are assigned to vids matching the bridge-domain on RPD.
RPD view
VLAN/interface configs of interest under routing-instance bridge-domains.
routing-instances { evpn-vxlan { ... bridge-domains { bd600 { vlan-id 600; --->bd600/vid interface ens3f2.600; vxlan { vni 2600; -> vxlan2600 } } bd601 { vlan-id 601; --->bd601/vid interface ens3f3.601; vxlan { vni 2601; -> vxlan2601 } } } } }
Kernel view
root@PE1_CRPD:/# bridge vlan show
port vlan ids ens3f2.600 600 PVID Egress Untagged ens3f3.601 601 PVID Egress Untagged __crpd-brd2 None vxlan2600 600 PVID Egress Untagged vxlan2601 601 PVID Egress Untagged irbbe-brd2 600 601
-
Verify if irb interface (vlan subinterface with bridge-domains vlan-id) is
created in kernel corresponding to the routing-interface configuration under
bridge-domains.
RPD view
IRB interface configs of interest under routing-instance bridge-domains.
routing-instances { evpn-vxlan { ... bridge-domains { bd600 { vlan-id 600; routing-interface irb.600; } bd601 { vlan-id 601; routing-interface irb.601; } } } }
Kernel view
root@PE1_CRPD:/# ip -d link show irb.600
20: irb.600@irbve-brd2: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master __crpd-vrf1 state UP mode DEFAULT group default qlen 1000 link/ether d6:a3:f9:94:70:78 brd ff:ff:ff:ff:ff:ff promiscuity 0 vlan protocol 802.1Q id 600 <REORDER_HDR> vrf_slave table 1 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
root@PE1_CRPD:/# ip -d link show irb.601
22: irb.601@irbve-brd2: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master __crpd-vrf1 state UP mode DEFAULT group default qlen 1000 link/ether d6:a3:f9:94:70:78 brd ff:ff:ff:ff:ff:ff promiscuity 0 vlan protocol 802.1Q id 601 <REORDER_HDR> vrf_slave table 1 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
-
Ensure if ipv4/ipv6 addresses are assigned to the irb interfaces.
RPD view
IP address configs of IRB interfaces.
interfaces { irb { unit 600 { family inet { address 99.60.0.254/24; } family inet6 { address 1234::99.60.0.254/120; } } unit 601 { family inet { address 99.60.1.254/24; } family inet6 { address 1234::99.60.1.254/120; } } } }
Kernel view
root@PE1_CRPD:/# ip addr show irb.600
20: irb.600@irbve-brd2: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master __crpd-vrf1 state UP group default qlen 1000 link/ether d6:a3:f9:94:70:78 brd ff:ff:ff:ff:ff:ff inet 99.60.0.254/24 scope global irb.600 valid_lft forever preferred_lft forever inet6 1234::633c:fe/120 scope global valid_lft forever preferred_lft forever inet6 fe80::d4a3:f9ff:fe94:7078/64 scope link valid_lft forever preferred_lft forever
root@PE1_CRPD:/# ip addr show irb.601
22: irb.601@irbve-brd2: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master __crpd-vrf1 state UP group default qlen 1000 link/ether d6:a3:f9:94:70:78 brd ff:ff:ff:ff:ff:ff inet 99.60.1.254/24 scope global irb.601 valid_lft forever preferred_lft forever inet6 1234::633c:1fe/120 scope global valid_lft forever preferred_lft forever inet6 fe80::d4a3:f9ff:fe94:7078/64 scope link valid_lft forever preferred_lft forever
-
Verify bridge flood entries are created in kernel, correpsonding to the
received IM (inclusive multicast) route entries received from peers.
RPD view
root@PE1_CRPD> show route table evpn-vxlan.evpn.0 protocol bgp | grep IM
3:81.2.2.2:1::2600::81.2.2.2/248 IM 3:81.2.2.2:1::2601::81.2.2.2/248 IM
Kernel view
root@PE1_CRPD:/# bridge fdb show br __crpd-brd2 state static | grep 00:00:00:00:00:00
00:00:00:00:00:00 dev vxlan2600 dst 81.2.2.2 self static 00:00:00:00:00:00 dev vxlan2601 dst 81.2.2.2 self static
-
Verify local MAC entries are learnt and advertised by EVPN to remote
peers.
RPD view
root@PE1_CRPD> show evpn database instance evpn-vxlan origin local
Instance: evpn-vxlan VLAN DomainId MAC address Active source Timestamp IP address 2600 00:11:11:11:60:00 ens3f2.600 May 10 23:49:46 99.60.0.1 1234::633c:1 fe80::5668:a302:5854:1f14 2600 d6:a3:f9:94:70:78 irb.600 Apr 29 21:08:59 99.60.0.254 1234::633c:fe fe80::d4a3:f9ff:fe94:7078 2601 00:11:11:11:60:10 ens3f3.601 May 10 23:47:44 99.60.1.1 1234::633c:101 fe80::5668:a302:5954:1f15 2601 d6:a3:f9:94:70:78 irb.601 Apr 29 21:08:59 99.60.1.254 1234::633c:1fe fe80::d4a3:f9ff:fe94:7078
root@PE1_CRPD> show route table evpn-vxlan.evpn.0 protocol evpn | grep MAC
2:81.1.1.1:1::2600::00:11:11:11:60:00/304 MAC/IP 2:81.1.1.1:1::2601::00:11:11:11:60:10/304 MAC/IP 2:81.1.1.1:1::2600::00:11:11:11:60:00::99.60.0.1/304 MAC/IP 2:81.1.1.1:1::2601::00:11:11:11:60:10::99.60.1.1/304 MAC/IP 2:81.1.1.1:1::2600::00:11:11:11:60:00::1234::633c:1/304 MAC/IP 2:81.1.1.1:1::2600::00:11:11:11:60:00::fe80::5668:a302:5854:1f14/304 MAC/IP 2:81.1.1.1:1::2601::00:11:11:11:60:10::1234::633c:101/304 MAC/IP 2:81.1.1.1:1::2601::00:11:11:11:60:10::fe80::5668:a302:5954:1f15/304 MAC/IP
Kernel view
Mac entries are learnt from bridge fdb table
root@PE1_CRPD:/# bridge fdb show br __crpd-brd2 brport ens3f2.600 state dynamic
00:11:11:11:60:00 vlan 600 master __crpd-brd2
root@PE1_CRPD:/# bridge fdb show br __crpd-brd2 brport ens3f3.601 state dynamic
00:11:11:11:60:10 vlan 601 master __crpd-brd2
Mac+ip bindings are learnt from ip neigh table
root@PE1_CRPD:/# ip neigh show dev irb.600 | grep -v PERMANENT
99.60.0.1 lladdr 00:11:11:11:60:00 REACHABLE 1234::633c:1 lladdr 00:11:11:11:60:00 router STALE fe80::5668:a302:5854:1f14 lladdr 00:11:11:11:60:00 router STALE
root@PE1_CRPD:/# ip neigh show dev irb.601 | grep -v PERMANENT
99.60.1.1 lladdr 00:11:11:11:60:10 REACHABLE 1234::633c:101 lladdr 00:11:11:11:60:10 router STALE fe80::5668:a302:5954:1f15 lladdr 00:11:11:11:60:10 router STALE
-
Verify remote MAC entries are learnt and programmed to kernel.
RPD view
root@PE1_CRPD> show route table evpn-vxlan.evpn.0 protocol bgp | grep MAC
2:81.2.2.2:1::2600::00:22:22:22:60:00/304 MAC/IP 2:81.2.2.2:1::2601::00:22:22:22:60:10/304 MAC/IP 2:81.2.2.2:1::2600::00:22:22:22:60:00::99.60.0.2/304 MAC/IP 2:81.2.2.2:1::2601::00:22:22:22:60:10::99.60.1.2/304 MAC/IP 2:81.2.2.2:1::2600::00:22:22:22:60:00::1234::633c:2/304 MAC/IP 2:81.2.2.2:1::2600::00:22:22:22:60:00::fe80::5668:a302:5854:1f09/304 MAC/IP 2:81.2.2.2:1::2601::00:22:22:22:60:10::1234::633c:102/304 MAC/IP 2:81.2.2.2:1::2601::00:22:22:22:60:10::fe80::5668:a302:5954:1f0a/304 MAC/IP
root@PE1_CRPD> show evpn database instance evpn-vxlan origin remote
Instance: evpn-vxlan VLAN DomainId MAC address Active source Timestamp IP address 2600 00:22:22:22:60:00 81.2.2.2 Apr 29 23:51:56 99.60.0.2 1234::633c:2 fe80::5668:a302:5854:1f09 2601 00:22:22:22:60:10 81.2.2.2 Apr 29 23:51:56 99.60.1.2 1234::633c:102 fe80::5668:a302:5954:1f0a
Kernel view
Macs are programmed to bridge fdb table in Linux
root@PE1_CRPD:/# bridge fdb show br __crpd-brd2 dev vxlan2600 state static
00:22:22:22:60:00 vlan 600 master __crpd-brd193 static 00:00:00:00:00:00 dst 81.2.2.2 self static 00:22:22:22:60:00 dst 81.2.2.2 self static
root@PE1_CRPD:/# bridge fdb show br __crpd-brd2 dev vxlan2601 state static
00:22:22:22:60:10 vlan 601 master __crpd-brd193 static 00:00:00:00:00:00 dst 81.2.2.2 self static 00:22:22:22:60:10 dst 81.2.2.2 self static
Mac+ip bindings are programmed to ip neigh table
root@PE1_CRPD:/# ip neigh show dev irb.600 | grep PERMANENT
99.60.0.2 lladdr 00:22:22:22:60:00 PERMANENT fe80::5668:a302:5854:1f09 lladdr 00:22:22:22:60:00 PERMANENT 1234::633c:2 lladdr 00:22:22:22:60:00 PERMANENT
root@PE1_CRPD:/# ip neigh show dev irb.601 | grep PERMANENT
99.60.1.2 lladdr 00:22:22:22:60:10 PERMANENT fe80::5668:a302:5954:1f0a lladdr 00:22:22:22:60:10 PERMANENT 1234::633c:102 lladdr 00:22:22:22:60:10 PERMANENT