Troubleshoot using the vRouter CLI
vif, dpdkinfo, flow,
rt, nh commands.
Accessing the vRouter CLI
Refer to Access vRouter CLI to learn how to access the vRouter CLI.
Troubleshooting via the vRouter CLI
You can run commands in the CLI to learn about the state of the vRouter.
Verify vRouter
Interfaces via the vif Command
Here is the list of options
available for the vif
command.
bash-5.1# vif --help
Usage: vif [--create <intf_name> --mac <mac>]
[--add <intf_name> --mac <mac> --vrf <vrf>
--type [vhost|agent|physical|virtual|monitoring|workload|lt]
--transport [eth|pmd|virtual|socket]
--xconnect <physical interface name>
--policy, --vhost-phys, --dhcp-enable]
--vif <vif ID> --id <intf_id> --parent_id <intf_id>
--vlan <vlan_id> --pmd --pci] --qos-map <qos_map_id>]
[--delete <intf_id>|<intf_name>]
[--get <intf_id>][--kernel][--core <core number>][--rate] [--get-drop-stats]
[--set <intf_id> --vlan <vlan_id> --vrf <vrf_id>]
[--list][--core <core number>][--rate]
[--clear][--id <intf_id>][--core <core_number>]
[--l2stats] [--list]
[--l2stats] [--get <intf_id>][--vlan <vlan_id>]
[--l2stats] [--clear][--id <intf_id>][--vlan <vlan_id>]
[--sock-dir <sock dir>]
[--vhostsock-dir <sock dir>]
[--vhostsock-filename <sock filename>]
[--get-features --id <intf_id>]
[--help]
bash-5.1#The command shown below allows you to see which interfaces are present on the vRouter:
vif --list
Vrouter Interface Table
Flags: P=Policy, X=Cross Connect, S=Service Chain, Mr=Receive Mirror
Mt=Transmit Mirror, Tc=Transmit Checksum Offload, L3=Layer 3, L2=Layer 2
D=DHCP, Vp=Vhost Physical, Pr=Promiscuous, Vnt=Native Vlan Tagged
Mnp=No MAC Proxy, Dpdk=DPDK PMD Interface, Rfl=Receive Filtering Offload, Mon=Interface is Monitored
Uuf=Unknown Unicast Flood, Vof=VLAN insert/strip offload, Df=Drop New Flows, L=MAC Learning Enabled
Proxy=MAC Requests Proxied Always, Er=Etree Root, Mn=Mirror without Vlan Tag, HbsL=HBS Left Intf
HbsR=HBS Right Intf, Ig=Igmp Trap Enabled, Ml=MAC-IP Learning Enabled, Me=Multicast Enabled
vif0/0 Socket: unix
Type:Agent HWaddr:00:00:5e:00:01:00
Vrf:65535 Flags:L2 QOS:-1 Ref:3
RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 0
RX packets:0 bytes:0 errors:0
TX packets:11 bytes:4169 errors:0
Drops:0
vif0/1 PCI: 0000:00:00.0 (Speed 25000, Duplex 1)
Type:Physical HWaddr:46:37:1f:de:df:bc
Vrf:65535 Flags:L2Vof QOS:-1 Ref:8
RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 0
Fabric Interface: eth_bond_bond0 Status: UP Driver: net_bonding
Slave Interface(0): 0000:3b:02.0 Status: UP Driver: net_iavf
Slave Interface(1): 0000:3b:02.1 Status: UP Driver: net_iavf
Vlan Mode: Trunk Vlan: 100 200 300 700-705
RX packets:0 bytes:0 errors:0
TX packets:378 bytes:81438 errors:0
Drops:0
vif0/2 PCI: 0000:3b:0a.0 (Speed 25000, Duplex 1)
Type:Workload HWaddr:ba:69:c0:b7:1f:ba
Vrf:0 Flags:L2Vof QOS:-1 Ref:7
RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 0
Fabric Interface: 0000:3b:0a.0 Status: UP Driver: net_iavf
Vlan Mode: Access Vlan Id: 700 OVlan Id: 700
RX packets:378 bytes:81438 errors:2
TX packets:0 bytes:0 errors:0
Drops:391View the running configuration of the vRouter
To see the status of the vRouter, enter the following command in the vRouter CLI:[root@jcnr-01 /]# ps -eaf | grep vrouter-dpdk root 116 90 99 Mar30 ? 118-08:05:37 /contrail-vrouter-dpdk --no-daemon --socket-mem=1024 1024 --allow=0000:5a:02.0 --vdev=eth_bond_bond0,mode=1,socket_id=0,mac=3a:1a:b7:86:1c:4f,primary=0000:5a:02.0, slave=0000:5a:02.0 --yield_option 0 --ddp root 1134749 1134365 0 16:41 pts/0 00:00:00 grep --color=auto vrouter-dpdk
The output contains several elements.
| Flag | Meaning |
|---|---|
--allow=<PCI Id> |
The PCI ID of fabric and fabric workload interfaces. More than one ID can appear in the output. These IDs serve as an allowlist. |
--ddp |
Enable Intel DDP support. We enable DDP by default in the values.yaml file in the vRouter. Note:
The Intel XL710 NIC does not support DDP. |
View status and statistics of DPDK using the dpdkinfo Command
The dpdkinfo command provides insight into the status and statistics of DPDK. The dpdkinfo command has many options. The following sections describe the available options and the example output from the dpdkinfo command. You can run the dpdkinfo command only from within the vRouter-agent CLI.
dpdkinfo --help
Usage: dpdkinfo [--help]
--version|-v Show DPDK Version
--bond|-b Show Master/Slave bond information
--lacp|-l <all/conf> Show LACP information from DPDK
--mempool|-m <all/<mempool-name>> Show Mempool information
--stats|-n <vif index value> Show Stats information
--xstats|-x <vif index value> Show Extended Stats information
--lcore|-c Show Lcore information
--app|-a Show App information
--ddp|-d <list> <list-flow> Show DDP information for X710 NIC
--rx_vlan|-z <value> Show VLan information
Optional: --buffsz <value> Send output buffer size (less than 1000Mb)The
command dpdkinfo -c shows the Lcores assigned to DPDK VF fabric
interfaces and the queue ID for each
interface.
dpdkinfo -c No. of forwarding lcores: 4 Lcore 10: Interface: 0000:18:01.1 Queue ID: 0 Interface: 0000:18:0d.1 Queue ID: 0 Interface: 0000:86:00.0 Queue ID: 0 Lcore 11: Interface: 0000:18:01.1 Queue ID: 1 Interface: 0000:18:0d.1 Queue ID: 1 Interface: 0000:86:00.0 Queue ID: 1 Lcore 12: Interface: 0000:18:01.1 Queue ID: 2 Interface: 0000:18:0d.1 Queue ID: 2 Interface: 0000:86:00.0 Queue ID: 2 Lcore 13: Interface: 0000:18:01.1 Queue ID: 3 Interface: 0000:18:0d.1 Queue ID: 3 Interface: 0000:86:00.0 Queue ID: 3
The command dpdkinfo -m
all shows all of the memory pool
information.
dpdkinfo -m all --------------------------------------------------- Name Size Used Available --------------------------------------------------- rss_mempool 16384 1549 14835 frag_direct_mempool 4096 0 4096 frag_indirect_mempool 4096 0 4096 packet_mbuf_pool 8192 2 8190
The command dpdkinfo -n
3 displays statistical information for a specific
interface.
dpdkinfo -n 3
Interface Info(0000:18:0d.1):
RX Device Packets:6710, Bytes:1367533, Errors:0, Nombufs:0
Dropped RX Packets:0
TX Device Packets:0, Bytes:0, Errors:0
Queue Rx:
Tx:
Rx Bytes:
Tx Bytes:
Errors:The command dpdkinfo -x 3 displays
extended statistical information for a specific
interface.
dpdkinfo -x 3 Driver Name:net_iavf Interface Info:0000:18:0d.1 Rx Packets: rx_good_packets: 6701 rx_unicast_packets: 0 rx_multicast_packets: 2987 rx_broadcast_packets: 3714 rx_dropped_packets: 0 Tx Packets: tx_good_packets: 0 tx_unicast_packets: 0 tx_multicast_packets: 0 tx_broadcast_packets: 0 tx_dropped_packets: 0 Rx Bytes: rx_good_bytes: 1365696 Tx Bytes: tx_good_bytes: 0 Errors: rx_missed_errors: 0 rx_errors: 0 tx_errors: 0 rx_mbuf_allocation_errors: 0 inline_ipsec_crypto_ierrors: 0 inline_ipsec_crypto_ierrors_sad_lookup: 0 inline_ipsec_crypto_ierrors_not_processed: 0 inline_ipsec_crypto_ierrors_icv_fail: 0 inline_ipsec_crypto_ierrors_length: 0 Others: inline_ipsec_crypto_ipackets: 0 ---------------------------------------------------------------------
The
command dpdkinfo -m debug verifies the hugepage size in use by the vRouter.
The output displays the memory zone table with a HugepageSz column that
shows the hugepage size allocated to each zone.
For a vRouter using 2MB hugepages, the
HugepageSz column shows 2M for all
entries:
Zone name Size HugepageSz SockID Virtual IOVA Flags rte_eth_dev_data 393280 2M 0 0x222019cd40 0x1cd39cd40 0x0 rte_mbuf_dyn 192 2M 0 0x222019cb40 0x1cd39cb40 0x0 RG_HT_fdir_0000:d8:00.1 262528 2M 0 0x22201562c0 0x1cd3562c0 0x0 MP_rss_mempool 2113792 2M 0 0x221fcd0040 0x1756d0040 0x0 MP_rss_mempool_0 545408704 2M 0 0x22203db940 0x1cd1db940 0x0 MP_packet_mbuf_pool 2113792 2M 0 0x221aea3c00 0x225ca3c00 0x0 MP_packet_mbuf_pool_0 17320000 2M 0 0x2219dff140 0x2263ff140 0x0 ...
For a vRouter using 1G hugepages, the HugepageSz column
shows 1G for all entries.
- Troubleshooting L2 Configuration and Statistics
- Troubleshooting L3 Configuration and Statistics
- Troubleshooting CoS Statistics via qosdpdk
Troubleshooting L2 Configuration and Statistics
View L2 Configuration and Statistics
-
Here is the list of options for the
bdcommand.bash-5.1# bd --help Usage: bd --dump bd --get --vrf <vrf_id> --vlan <vlan_id> bd --help --dump Dumps the bd table --get Dumps the bd entry corresponding to vlan and vrf --sock-dir <netlink sock dir> --help Prints this help message - Verify the bridge domain information on vRouter using the
bdcommand:bash-5.1# bd Usage: bd --dump bd --get --vrf <vrf_id> --vlan <vlan_id> bd --help --dump Dumps the bd table --get Dumps the bd entry corresponding to vlan and vrf --sock-dir <netlink sock dir> --help Prints this help messagebash-5.1# bd --dump Bridge Domain (BD) Table ----------------------------------------------------------------------- VRF VLAN BD BD Name ----------------------------------------------------------------------- 1 100 1 vswitch_bd100 1 101 2 vswitch_bd101 2 100 3 vs1_bd100 2 200 4 vs1_bd200 -----------------------------------------------------------------------bash-5.1# bd --get --vrf 1 --vlan 100 Bridge Domain (BD) Table ----------------------------------------------------------------------- VRF VLAN BD BD Name ----------------------------------------------------------------------- 1 100 1 vswitch_bd100 -----------------------------------------------------------------------
- The command below shows the MAC addresses that the vRouter has dynamically learned.
Broadcast route is created for each bridge domain. This command also shows the broadcast
route's next
hop.
bash-5.1# rt --dump 1 --family bridge Flags: L=Label Valid, Df=DHCP flood, Mm=Mac Moved, L2c=L2 Evpn Control Word, N=New Entry, Ec=EvpnControlProcessing vRouter bridge table 0/1 Index BdID DestMac Flags Label/VNID Nexthop Stats 27284 1 0:31:4:64:0:2 Df - 23 3 72352 1 0:31:4:64:0:1 Df - 16 4 100676 1 ff:ff:ff:ff:ff:ff LDf 0 26 24 140472 2 ff:ff:ff:ff:ff:ff LDf 0 31 12 214108 2 0:31:4:65:0:1 Df - 28 3
- From the command output above, broadcast route in bridge domain (BD) 2 points to next
hop 31. You can get more details about the next hop using the
nh --get nhIDcommand.bash-5.1# nh --get 31 Id:31 Type:Composite Fmly:AF_BRIDGE Rid:0 Ref_cnt:2 Vrf:1 Next NH:-1 NH Label:0 NH Hit Count:12 Flags:Valid, Multicast, Etree Root, Src Validate, Sub NH(label): 30(0) Id:30 Type:Composite Fmly:AF_BRIDGE Rid:0 Ref_cnt:2 Vrf:1 Next NH:-1 NH Label:0 NH Hit Count:12 Flags:Valid, Encap, Etree Root, Sub NH(label): 7(0) 28(0) Id:7 Type:Encap Fmly:AF_BRIDGE Rid:0 Ref_cnt:4 Vrf:0 Next NH:-1 NH Label:0 NH Hit Count:37 Flags:Valid, Etree Root, EncapFmly:0806 Oif:1 Len:14 Encap Data: 6a a4 1d 45 d6 65 6a a4 1d 45 d6 65 Id:28 Type:Encap Fmly:AF_BRIDGE Rid:0 Ref_cnt:3 Vrf:1 Next NH:-1 NH Label:0 NH Hit Count:0 Flags:Valid, Mac Learn, Etree Root, EncapFmly:0806 Oif:6 Len:14 Encap Data: 00 31 04 65 00 01 00 31 04 65 00 01The highlighted
Oifvalues correspond tovif1andvif6interfaces that are a part of BD 2.
- The Layer 2 stats are available using the
vifutility.bash-5.1# vif --l2stats --get 4 --vlan 100 Vlan id count: 1 Statistics for vif 4 vlan 100 Rx Pkts Rx Bytes Tx Pkts Tx Bytes Unicast 47 4682 47 4682 Broadcast 0 0 0 0 Multicast 0 0 3 270 Flood 0 0 0 0
- Use the following command to clear L2
stats:
bash-5.1# vif --l2stats --clear --vlan 100
Packet Tracing via the dropstats Command
The vRouter tracks the packets that it drops and includes the reason for dropping them. The table below shows the common reasons for vRouter to drop a packet. When you execute the dropstats command, the vRouter does not show a counter if the count for that counter is 0.
| Counter Name | Meaning |
|---|---|
L2 bd table drop |
No interfaces in bridge domain |
L2 untag pkt drop |
Untagged packet arrives on trunk or sub-interface |
L2 Invalid Vlan |
Packet VLAN does not match interface VLAN |
L2 Mac Table Full |
No more entries available in the MAC table |
L2 ACL drop |
Packet matched firewall filter (ACL) drop rule |
L2 Src Mac lookup fail |
Unable to match (or learn) the source MAC address |
Example output from the dropstats command looks like:
dropstats L2 bd table Drop 43 L2 untag pkt drop 716 L2 Invalid Vlan 7288253 Rate limit exceeded 673179706 L2 Mac Table Full 41398787 L2 ACL drop 8937037 L2 Src Mac lookup fail 247046
Troubleshooting L3 Configuration and Statistics
Display routes and next hops using the rt and
nh Commands for L3 Deployments
Here is the list of options available for the rt command.
bash-5.1# rt --help
Usage: rt --dump <vrf_id> [--family <inet|inet6|bridge|ccc>]>
rt --dump <vrf_id/bd_id> --family bridge
rt --get <address/plen> --vrf <id> [--family <inet|inet6|ccc>]
rt --get <address/bd_id> --vrf <id> --family bridge
rt --family bridge --show-mac-move
rt --monitor
rt --help
--dump Dumps the routing table corresponding to vrf_id
--family Optional family specification to --dump command
Specification should be one of "inet" or "bridge"
--show-mac-move Shows the MAC move detection table (bridge family only)
--monitor Watch for netlink broadcasted messages
--sock-dir <netlink sock dir>
--help Prints this help messageFor example, for IPv4 traffic:
rt --get 172.68.20.2/32 --vrf 4 Match 172.68.20.2/32 in vRouter inet4 table 0/4/unicast Flags: L=Label Valid, P=Proxy ARP, T=Trap ARP, F=Flood ARP, Ml=MAC-IP learnt route vRouter inet4 routing table 0/4/unicast Destination PPL Flags Label Nexthop Stitched MAC(Index) 172.68.20.2/32 0 LPT 16 193 -
nh --get 193 Id:193 Type:Tunnel Fmly: AF_INET Rid:0 Ref_cnt:264 Vrf:0 Flags:Valid, Policy, MPLSoUDP, Etree Root, Oif:4 Len:14 Data:88 e6 4b 09 7d 46 40 a6 b7 2c a4 48 08 00 Sip:1.1.1.35 Dip:1.1.24.24
For example, for IPv6 traffic:
rt --get 2001:172:68:20::/64 --vrf 4 --family inet6 Match 2001:172:68:20::/64 in vRouter inet6 table 0/4/unicast Flags: L=Label Valid, P=Proxy ARP, T=Trap ARP, F=Flood ARP, Ml=MAC-IP learnt route vRouter inet6 routing table 0/4/unicast Destination PPL Flags Label Nexthop Stitched MAC(Index) 2001:172:68:20::/64 0 LPT 16 193 -
nh --get 193 Id:193 Type:Tunnel Fmly: AF_INET Rid:0 Ref_cnt:264 Vrf:0 Flags:Valid, Policy, MPLSoUDP, Etree Root, Oif:4 Len:14 Data:88 e6 4b 09 7d 46 40 a6 b7 2c a4 48 08 00 Sip:1.1.1.35 Dip:1.1.24.24
Display all active flows using the flow Command
for L3 Deployment
Use the flow command to display all active flows in a system. For
example:
flow -l --match 169.83.47.170:9398
Flow table(size 161218560, entries 629760)
Entries: Created 162630 Added 162614 Deleted 35136 Changed 35202Processed 162630 Used Overflow entries 0
(Created Flows/CPU: 0 0 0 0 0 0 0 0 0 0 241 546 15 161828)(oflows 0)
Action:F=Forward, D=Drop N=NAT(S=SNAT, D=DNAT, Ps=SPAT, Pd=DPAT, L=Link Local Port)
Other:K(nh)=Key_Nexthop, S(nh)=RPF_Nexthop
Flags:E=Evicted, Ec=Evict Candidate, N=New Flow, M=Modified Dm=Delete Marked
TCP(r=reverse):S=SYN, F=FIN, R=RST, C=HalfClose, E=Established, D=Dead
Stats:Packets/Bytes
Listing flows matching ([169.83.47.170]:9398)
Index Source:Port/Destination:Port Proto(V)
-----------------------------------------------------------------------------------
328196<=>524233 169.83.47.170:9398 6 (2)
172.68.20.20:2159
(Gen: 3, K(nh):206, Action:F, Flags:, TCP:, E:1, QOS:-1, S(nh):206, Stats:6/360,
SPort 63929, TTL 0, Sinfo 38.0.0.0)
524233<=>328196 172.68.20.20:2159 6 (2)
169.83.47.170:9398
(Gen: 3, K(nh):206, Action:F, Flags:, TCP:, QOS:-1, S(nh):250, Stats:0/0,
SPort 60311, TTL 0, Sinfo 0.0.0.0)
Troubleshooting CoS Statistics via qosdpdk
The qosdpdk utility provides per-forwarding-lcore CoS statistics
directly from the vRouter DPDK datapath. Use it for granular operational troubleshooting
when aggregated Prometheus or gNMI data does not pinpoint the affected lcore. To run
qosdpdk commands interactively, open a shell in the
contrail-vrouter-agent-dpdk container:
kubectl exec -it -n contrail <vRouter-dpdk-pod-name> -c contrail-vrouter-agent-dpdk -- bash
To view the usage and examples of the qosdpdk utility, run
qosdpdk --help:
bash-5.1# qosdpdk --help Usage: qosdpdk [--get <classifier/cla> --name <NAME> --tc <dscp/dscpv6>] [--get <re-write/rw> --name <NAME> --tc <dscp/dscpv6/exp>] [--get <policer/pol> --name <NAME> [--core <core number>] [--get <scheduler/sch> --name <NAME> [--get <dropper/dro> --name <NAME> [--list <cla/rw/pol/sch/dro>] [--core <core number>] [--stat <ba_cla/mf_cla/rw>] [--tc <dscp/dscpv6/exp>] [--core <core number>] [--sock-dir <sock dir>] [--help] Examples - qosdpdk --get classifier --name cos1 --tc dscp qosdpdk --get re-write --name dot1 --tc dscpv6 qosdpdk --get policer --name policer1 qosdpdk --get scheduler --name scheduler1 qosdpdk --get dropper --name dropper1 qosdpdk --get pol --name policer1 qosdpdk --get cla --tc dscp --name cla1 qosdpdk --get rw --tc exp --name exp1 qosdpdk --get sch --name sch1 qosdpdk --get dro --name dp1 qosdpdk --list cla qosdpdk --list re-write qosdpdk --list sch qosdpdk --stat ba_cla
|
Option |
Arguments |
Description |
|---|---|---|
|
|
|
Retrieve statistics for a specific named CoS object. The output varies by module:
|
|
|
Optional: |
List all CoS objects configured for the specified module. The output returns the
names and identifiers of every configured object of that type (for example, all
classifier entries, all policers, or all scheduler entries), providing an
inventory you can use to identify valid names for use with the
|
|
|
Optional: Optional: |
Display aggregated statistics per forwarding lcore for BA classifier, MF
classifier, or rewrite. When |
|
|
|
Override the default DPDK socket directory. |
|
|
nil |
Display usage information and examples. |
The following examples show sample qosdpdk command output for BA
classifier and rewrite statistics.
BA classifier statistics per forwarding lcore (DSCP traffic class):
bash-5.1# qosdpdk --stat ba_cla --tc dscp --core 1 ------------------------------------------------------------------------------------------------- | Type | core Id | Traffic Class Type | Forwarding Class | Packets | Bytes | ------------------------------------------------------------------------------------------------- | BA classifier | 15 | dscp | fc0(64) | 0 | 0 | | | | | fc1(65) | 0 | 0 | | | | | fc2(66) | 0 | 0 | | | | | fc3(67) | 0 | 0 | | | | | fc4(68) | 0 | 0 | | | | | fc5(69) | 0 | 0 | | | | | fc6(70) | 0 | 0 | | | | | fc7(71) | 0 | 0 | -------------------------------------------------------------------------------------------------
The --core argument specifies the DPDK lcore index, not the physical CPU
core ID. The core ID shown in the output reflects the actual CPU core assigned to that
lcore by DPDK.
Rewrite statistics for all lcores (all traffic class types):
bash-5.1# qosdpdk --stat rw ------------------------------------------------------------------------------------------------- | Type | core Id | Traffic Class Type | Forwarding Class | Packets | Bytes | ------------------------------------------------------------------------------------------------- | Rewrite | 15 | dscp | fc1(71) | 0 | 0 | | | | | fc2(75) | 0 | 0 | | | | | fc3(77) | 0 | 0 | ------------------------------------------------------------------------------------------------- | Rewrite | 16 | dscp | fc1(71) | 1 | 54 | | | | | fc2(75) | 1 | 54 | | | | | fc3(77) | 1 | 54 | ------------------------------------------------------------------------------------------------- | Rewrite | 15 | dscpv6 | fc1(71) | 0 | 0 | | | | | fc2(75) | 0 | 0 | | | | | fc3(77) | 0 | 0 | ------------------------------------------------------------------------------------------------- | Rewrite | 16 | dscpv6 | fc1(71) | 0 | 0 | | | | | fc2(75) | 0 | 0 | | | | | fc3(77) | 0 | 0 | ------------------------------------------------------------------------------------------------- | Rewrite | 15 | exp | fc1(71) | 0 | 0 | | | | | fc2(75) | 0 | 0 | | | | | fc3(77) | 0 | 0 | ------------------------------------------------------------------------------------------------- | Rewrite | 16 | exp | fc1(71) | 0 | 0 | | | | | fc2(75) | 0 | 0 | | | | | fc3(77) | 0 | 0 | -------------------------------------------------------------------------------------------------