Troubleshoot using the vRouter CLI
vif, dpdkinfo, flow,
rt, nh commands.
Accessing the vRouter CLI
Refer to Access vRouter CLI to learn how to access the vRouter CLI.
Troubleshooting via the vRouter CLI
You can run commands in the CLI to learn about the state of the vRouter.
Verify vRouter
Interfaces via the vif Command
Here is the list of options
available for the vif
command.
bash-5.1# vif --help
Usage: vif [--create <intf_name> --mac <mac>]
[--add <intf_name> --mac <mac> --vrf <vrf>
--type [vhost|agent|physical|virtual|monitoring|workload|lt]
--transport [eth|pmd|virtual|socket]
--xconnect <physical interface name>
--policy, --vhost-phys, --dhcp-enable]
--vif <vif ID> --id <intf_id> --parent_id <intf_id>
--vlan <vlan_id> --pmd --pci] --qos-map <qos_map_id>]
[--delete <intf_id>|<intf_name>]
[--get <intf_id>][--kernel][--core <core number>][--rate] [--get-drop-stats]
[--set <intf_id> --vlan <vlan_id> --vrf <vrf_id>]
[--list][--core <core number>][--rate]
[--clear][--id <intf_id>][--core <core_number>]
[--l2stats] [--list]
[--l2stats] [--get <intf_id>][--vlan <vlan_id>]
[--l2stats] [--clear][--id <intf_id>][--vlan <vlan_id>]
[--sock-dir <sock dir>]
[--vhostsock-dir <sock dir>]
[--vhostsock-filename <sock filename>]
[--get-features --id <intf_id>]
[--help]
bash-5.1#The command shown below allows you to see which interfaces are present on the vRouter:
vif --list
Vrouter Interface Table
Flags: P=Policy, X=Cross Connect, S=Service Chain, Mr=Receive Mirror
Mt=Transmit Mirror, Tc=Transmit Checksum Offload, L3=Layer 3, L2=Layer 2
D=DHCP, Vp=Vhost Physical, Pr=Promiscuous, Vnt=Native Vlan Tagged
Mnp=No MAC Proxy, Dpdk=DPDK PMD Interface, Rfl=Receive Filtering Offload, Mon=Interface is Monitored
Uuf=Unknown Unicast Flood, Vof=VLAN insert/strip offload, Df=Drop New Flows, L=MAC Learning Enabled
Proxy=MAC Requests Proxied Always, Er=Etree Root, Mn=Mirror without Vlan Tag, HbsL=HBS Left Intf
HbsR=HBS Right Intf, Ig=Igmp Trap Enabled, Ml=MAC-IP Learning Enabled, Me=Multicast Enabled
vif0/0 Socket: unix
Type:Agent HWaddr:00:00:5e:00:01:00
Vrf:65535 Flags:L2 QOS:-1 Ref:3
RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 0
RX packets:0 bytes:0 errors:0
TX packets:11 bytes:4169 errors:0
Drops:0
vif0/1 PCI: 0000:00:00.0 (Speed 25000, Duplex 1)
Type:Physical HWaddr:46:37:1f:de:df:bc
Vrf:65535 Flags:L2Vof QOS:-1 Ref:8
RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 0
Fabric Interface: eth_bond_bond0 Status: UP Driver: net_bonding
Slave Interface(0): 0000:3b:02.0 Status: UP Driver: net_iavf
Slave Interface(1): 0000:3b:02.1 Status: UP Driver: net_iavf
Vlan Mode: Trunk Vlan: 100 200 300 700-705
RX packets:0 bytes:0 errors:0
TX packets:378 bytes:81438 errors:0
Drops:0
vif0/2 PCI: 0000:3b:0a.0 (Speed 25000, Duplex 1)
Type:Workload HWaddr:ba:69:c0:b7:1f:ba
Vrf:0 Flags:L2Vof QOS:-1 Ref:7
RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 0
Fabric Interface: 0000:3b:0a.0 Status: UP Driver: net_iavf
Vlan Mode: Access Vlan Id: 700 OVlan Id: 700
RX packets:378 bytes:81438 errors:2
TX packets:0 bytes:0 errors:0
Drops:391View the running configuration of the vRouter
To see the status of the vRouter, enter the following command in the vRouter CLI:[root@jcnr-01 /]# ps -eaf | grep vrouter-dpdk root 116 90 99 Mar30 ? 118-08:05:37 /contrail-vrouter-dpdk --no-daemon --socket-mem=1024 1024 --allow=0000:5a:02.0 --vdev=eth_bond_bond0,mode=1,socket_id=0,mac=3a:1a:b7:86:1c:4f,primary=0000:5a:02.0, slave=0000:5a:02.0 --yield_option 0 --ddp root 1134749 1134365 0 16:41 pts/0 00:00:00 grep --color=auto vrouter-dpdk
The output contains several elements.
| Flag | Meaning |
|---|---|
--allow=<PCI Id> |
The PCI ID of fabric and fabric workload interfaces. More than one ID can appear in the output. These IDs serve as an allowlist. |
--ddp |
Enable Intel DDP support. We enable DDP by default in the values.yaml file in the vRouter. Note:
The Intel XL710 NIC does not support DDP. |
View status and statistics of DPDK using the dpdkinfo Command
The dpdkinfo command provides insight into the status and statistics of DPDK. The dpdkinfo command has many options. The following sections describe the available options and the example output from the dpdkinfo command. You can run the dpdkinfo command only from within the vRouter-agent CLI.
dpdkinfo --help
Usage: dpdkinfo [--help]
--version|-v Show DPDK Version
--bond|-b Show Master/Slave bond information
--lacp|-l <all/conf> Show LACP information from DPDK
--mempool|-m <all/<mempool-name>> Show Mempool information
--stats|-n <vif index value> Show Stats information
--xstats|-x <vif index value> Show Extended Stats information
--lcore|-c Show Lcore information
--app|-a Show App information
--ddp|-d <list> <list-flow> Show DDP information for X710 NIC
--rx_vlan|-z <value> Show VLan information
Optional: --buffsz <value> Send output buffer size (less than 1000Mb)The
command dpdkinfo -c shows the Lcores assigned to DPDK VF fabric
interfaces and the queue ID for each
interface.
dpdkinfo -c No. of forwarding lcores: 4 Lcore 10: Interface: 0000:18:01.1 Queue ID: 0 Interface: 0000:18:0d.1 Queue ID: 0 Interface: 0000:86:00.0 Queue ID: 0 Lcore 11: Interface: 0000:18:01.1 Queue ID: 1 Interface: 0000:18:0d.1 Queue ID: 1 Interface: 0000:86:00.0 Queue ID: 1 Lcore 12: Interface: 0000:18:01.1 Queue ID: 2 Interface: 0000:18:0d.1 Queue ID: 2 Interface: 0000:86:00.0 Queue ID: 2 Lcore 13: Interface: 0000:18:01.1 Queue ID: 3 Interface: 0000:18:0d.1 Queue ID: 3 Interface: 0000:86:00.0 Queue ID: 3
The command dpdkinfo -m
all shows all of the memory pool
information.
dpdkinfo -m all --------------------------------------------------- Name Size Used Available --------------------------------------------------- rss_mempool 16384 1549 14835 frag_direct_mempool 4096 0 4096 frag_indirect_mempool 4096 0 4096 packet_mbuf_pool 8192 2 8190
The command dpdkinfo -n
3 displays statistical information for a specific
interface.
dpdkinfo -n 3
Interface Info(0000:18:0d.1):
RX Device Packets:6710, Bytes:1367533, Errors:0, Nombufs:0
Dropped RX Packets:0
TX Device Packets:0, Bytes:0, Errors:0
Queue Rx:
Tx:
Rx Bytes:
Tx Bytes:
Errors:The command dpdkinfo -x 3 displays
extended statistical information for a specific
interface.
dpdkinfo -x 3 Driver Name:net_iavf Interface Info:0000:18:0d.1 Rx Packets: rx_good_packets: 6701 rx_unicast_packets: 0 rx_multicast_packets: 2987 rx_broadcast_packets: 3714 rx_dropped_packets: 0 Tx Packets: tx_good_packets: 0 tx_unicast_packets: 0 tx_multicast_packets: 0 tx_broadcast_packets: 0 tx_dropped_packets: 0 Rx Bytes: rx_good_bytes: 1365696 Tx Bytes: tx_good_bytes: 0 Errors: rx_missed_errors: 0 rx_errors: 0 tx_errors: 0 rx_mbuf_allocation_errors: 0 inline_ipsec_crypto_ierrors: 0 inline_ipsec_crypto_ierrors_sad_lookup: 0 inline_ipsec_crypto_ierrors_not_processed: 0 inline_ipsec_crypto_ierrors_icv_fail: 0 inline_ipsec_crypto_ierrors_length: 0 Others: inline_ipsec_crypto_ipackets: 0 ---------------------------------------------------------------------
Troubleshooting L2 Configuration and Statistics
View L2 Configuration and Statistics
-
Here is the list of options for the
bdcommand.bash-5.1# bd --help Usage: bd --dump bd --get --vrf <vrf_id> --vlan <vlan_id> bd --help --dump Dumps the bd table --get Dumps the bd entry corresponding to vlan and vrf --sock-dir <netlink sock dir> --help Prints this help message - Verify the bridge domain information on vRouter using the
bdcommand:bash-5.1# bd Usage: bd --dump bd --get --vrf <vrf_id> --vlan <vlan_id> bd --help --dump Dumps the bd table --get Dumps the bd entry corresponding to vlan and vrf --sock-dir <netlink sock dir> --help Prints this help messagebash-5.1# bd --dump Bridge Domain (BD) Table ----------------------------------------------------------------------- VRF VLAN BD BD Name ----------------------------------------------------------------------- 1 100 1 vswitch_bd100 1 101 2 vswitch_bd101 2 100 3 vs1_bd100 2 200 4 vs1_bd200 -----------------------------------------------------------------------bash-5.1# bd --get --vrf 1 --vlan 100 Bridge Domain (BD) Table ----------------------------------------------------------------------- VRF VLAN BD BD Name ----------------------------------------------------------------------- 1 100 1 vswitch_bd100 -----------------------------------------------------------------------
- The command below shows the MAC addresses that the vRouter has dynamically learned.
Broadcast route is created for each bridge domain. This command also shows the broadcast
route's next
hop.
bash-5.1# rt --dump 1 --family bridge Flags: L=Label Valid, Df=DHCP flood, Mm=Mac Moved, L2c=L2 Evpn Control Word, N=New Entry, Ec=EvpnControlProcessing vRouter bridge table 0/1 Index BdID DestMac Flags Label/VNID Nexthop Stats 27284 1 0:31:4:64:0:2 Df - 23 3 72352 1 0:31:4:64:0:1 Df - 16 4 100676 1 ff:ff:ff:ff:ff:ff LDf 0 26 24 140472 2 ff:ff:ff:ff:ff:ff LDf 0 31 12 214108 2 0:31:4:65:0:1 Df - 28 3
- From the command output above, broadcast route in bridge domain (BD) 2 points to next
hop 31. You can get more details about the next hop using the
nh --get nhIDcommand.bash-5.1# nh --get 31 Id:31 Type:Composite Fmly:AF_BRIDGE Rid:0 Ref_cnt:2 Vrf:1 Next NH:-1 NH Label:0 NH Hit Count:12 Flags:Valid, Multicast, Etree Root, Src Validate, Sub NH(label): 30(0) Id:30 Type:Composite Fmly:AF_BRIDGE Rid:0 Ref_cnt:2 Vrf:1 Next NH:-1 NH Label:0 NH Hit Count:12 Flags:Valid, Encap, Etree Root, Sub NH(label): 7(0) 28(0) Id:7 Type:Encap Fmly:AF_BRIDGE Rid:0 Ref_cnt:4 Vrf:0 Next NH:-1 NH Label:0 NH Hit Count:37 Flags:Valid, Etree Root, EncapFmly:0806 Oif:1 Len:14 Encap Data: 6a a4 1d 45 d6 65 6a a4 1d 45 d6 65 Id:28 Type:Encap Fmly:AF_BRIDGE Rid:0 Ref_cnt:3 Vrf:1 Next NH:-1 NH Label:0 NH Hit Count:0 Flags:Valid, Mac Learn, Etree Root, EncapFmly:0806 Oif:6 Len:14 Encap Data: 00 31 04 65 00 01 00 31 04 65 00 01The highlighted
Oifvalues correspond tovif1andvif6interfaces that are a part of BD 2.
- The Layer 2 stats are available using the
vifutility.bash-5.1# vif --l2stats --get 4 --vlan 100 Vlan id count: 1 Statistics for vif 4 vlan 100 Rx Pkts Rx Bytes Tx Pkts Tx Bytes Unicast 47 4682 47 4682 Broadcast 0 0 0 0 Multicast 0 0 3 270 Flood 0 0 0 0
- Use the following command to clear L2
stats:
bash-5.1# vif --l2stats --clear --vlan 100
Packet Tracing via the dropstats Command
The vRouter tracks the packets that it drops and includes the reason for dropping them. The table below shows the common reasons for vRouter to drop a packet. When you execute the dropstats command, the vRouter does not show a counter if the count for that counter is 0.
| Counter Name | Meaning |
|---|---|
L2 bd table drop |
No interfaces in bridge domain |
L2 untag pkt drop |
Untagged packet arrives on trunk or sub-interface |
L2 Invalid Vlan |
Packet VLAN does not match interface VLAN |
L2 Mac Table Full |
No more entries available in the MAC table |
L2 ACL drop |
Packet matched firewall filter (ACL) drop rule |
L2 Src Mac lookup fail |
Unable to match (or learn) the source MAC address |
Example output from the dropstats command looks like:
dropstats L2 bd table Drop 43 L2 untag pkt drop 716 L2 Invalid Vlan 7288253 Rate limit exceeded 673179706 L2 Mac Table Full 41398787 L2 ACL drop 8937037 L2 Src Mac lookup fail 247046
Troubleshooting L3 Configuration and Statistics
Display routes and next hops using the rt and
nh Commands for L3 Deployments
Here is the list of options available for the rt command.
bash-5.1# rt --help
Usage: rt --dump <vrf_id> [--family <inet|inet6|bridge|ccc>]>
rt --dump <vrf_id/bd_id> --family bridge
rt --get <address/plen> --vrf <id> [--family <inet|inet6|ccc>]
rt --get <address/bd_id> --vrf <id> --family bridge
rt --family bridge --show-mac-move
rt --monitor
rt --help
--dump Dumps the routing table corresponding to vrf_id
--family Optional family specification to --dump command
Specification should be one of "inet" or "bridge"
--show-mac-move Shows the MAC move detection table (bridge family only)
--monitor Watch for netlink broadcasted messages
--sock-dir <netlink sock dir>
--help Prints this help messageFor example, for IPv4 traffic:
rt --get 172.68.20.2/32 --vrf 4 Match 172.68.20.2/32 in vRouter inet4 table 0/4/unicast Flags: L=Label Valid, P=Proxy ARP, T=Trap ARP, F=Flood ARP, Ml=MAC-IP learnt route vRouter inet4 routing table 0/4/unicast Destination PPL Flags Label Nexthop Stitched MAC(Index) 172.68.20.2/32 0 LPT 16 193 -
nh --get 193 Id:193 Type:Tunnel Fmly: AF_INET Rid:0 Ref_cnt:264 Vrf:0 Flags:Valid, Policy, MPLSoUDP, Etree Root, Oif:4 Len:14 Data:88 e6 4b 09 7d 46 40 a6 b7 2c a4 48 08 00 Sip:1.1.1.35 Dip:1.1.24.24
For example, for IPv6 traffic:
rt --get 2001:172:68:20::/64 --vrf 4 --family inet6 Match 2001:172:68:20::/64 in vRouter inet6 table 0/4/unicast Flags: L=Label Valid, P=Proxy ARP, T=Trap ARP, F=Flood ARP, Ml=MAC-IP learnt route vRouter inet6 routing table 0/4/unicast Destination PPL Flags Label Nexthop Stitched MAC(Index) 2001:172:68:20::/64 0 LPT 16 193 -
nh --get 193 Id:193 Type:Tunnel Fmly: AF_INET Rid:0 Ref_cnt:264 Vrf:0 Flags:Valid, Policy, MPLSoUDP, Etree Root, Oif:4 Len:14 Data:88 e6 4b 09 7d 46 40 a6 b7 2c a4 48 08 00 Sip:1.1.1.35 Dip:1.1.24.24
Display all active flows using the flow Command
for L3 Deployment
Use the flow command to display all active flows in a system. For
example:
flow -l --match 169.83.47.170:9398
Flow table(size 161218560, entries 629760)
Entries: Created 162630 Added 162614 Deleted 35136 Changed 35202Processed 162630 Used Overflow entries 0
(Created Flows/CPU: 0 0 0 0 0 0 0 0 0 0 241 546 15 161828)(oflows 0)
Action:F=Forward, D=Drop N=NAT(S=SNAT, D=DNAT, Ps=SPAT, Pd=DPAT, L=Link Local Port)
Other:K(nh)=Key_Nexthop, S(nh)=RPF_Nexthop
Flags:E=Evicted, Ec=Evict Candidate, N=New Flow, M=Modified Dm=Delete Marked
TCP(r=reverse):S=SYN, F=FIN, R=RST, C=HalfClose, E=Established, D=Dead
Stats:Packets/Bytes
Listing flows matching ([169.83.47.170]:9398)
Index Source:Port/Destination:Port Proto(V)
-----------------------------------------------------------------------------------
328196<=>524233 169.83.47.170:9398 6 (2)
172.68.20.20:2159
(Gen: 3, K(nh):206, Action:F, Flags:, TCP:, E:1, QOS:-1, S(nh):206, Stats:6/360,
SPort 63929, TTL 0, Sinfo 38.0.0.0)
524233<=>328196 172.68.20.20:2159 6 (2)
169.83.47.170:9398
(Gen: 3, K(nh):206, Action:F, Flags:, TCP:, QOS:-1, S(nh):250, Stats:0/0,
SPort 60311, TTL 0, Sinfo 0.0.0.0)