Queue-Pair Hashing for RDMA Flows
Overview
Load balancing algorithms use hashing mechanisms to select the best outgoing interface from a link aggregation group (LAG) or ECMP bundle. Most load balancing algorithms hash (that is, divide the traffic load among links) based on 5-tuple information. The 5-tuple information covers Layer 2 (L2), which includes the source and destination MAC addresses; Layer 3 (L3), which includes the source and destination IP addresses; and Layer 4 (L4), which includes the TCP/UDP ports. However, in AI-ML training networks, most traffic uses Remote Direct Memory Access (RDMA) over Converged Ethernet version 2 (RoCEv2) for transport. ROCEv2 traffic might not have variance on L4 ports. This leads to less entropy and could result in less efficient load balancing.
In a network that uses RoCEv2 for application traffic transport, an RDMA connection sends traffic on a send queue and receives traffic on a receive queue. These queues form the RDMA connection. Together, the send queue and receive queue are referred to as a queue pair. Each queue pair has an identifiable prefix. ROCEv2 uses the destination queue pair as the flow identification field.
You can enhance load balancing on your AI-ML training network by including the destination queue pair in the hash calculation. This is called queue-pair hashing. Adding the destination queue pair to the hash calculation increases the entropy of the system and improves the overall load balancing efficiency. On devices and releases that support this feature, queue-pair hashing is enabled by default for all ROCEv2 traffic.
When you enable dynamic load balancing (DLB) on your device, queue-pair hashing is automatically disabled on link aggregation group (LAG) interfaces.
Configuration
Verify Queue-Pair Hashing
By default, the load balancing algorithm includes the RDMA queue pair as part of the hash calculation for RDMA traffic (both IPv4 and IPv6). No configuration is required.
To confirm the feature is enabled, use the following command and confirm that the “RDMA Queue Pair” field in the output is marked “yes”:
user@device> show forwarding-options enhanced-hash-key Current RTAG7 Settings ------------------------- Hash-Mode :layer2-payload Hash-Seed :108202401 inet RTAG7 settings: ---------------------- inet packet fields protocol :yes Destination IPv4 Addr :yes Source IPv4 Addr :yes destination L4 Port :yes Source L4 Port :yes Vlan id :no RDMA Queue Pair :yes inet non-packet fields incoming device :yes incoming port :yes inet6 RTAG7 settings: ---------------------- inet6 packet fields next-header :yes Destination IPv6 Addr :yes Source IPv6 Addr :yes destination L4 Port :yes Source L4 Port :yes Vlan id :no RDMA Queue Pair :yes inet6 non-packet fields incoming device :yes incoming port :yes Hash-Parameter Settings for ECMP: ------------------------------------ Hash Function = CRC16_BISYNC Hash offset base = 16 Hash offset = 4294967280 Hash preprocess = 0 Hash-Parameter Settings for LAG: ------------------------------------ Hash Function = CRC16_CCITT Hash offset base = 0 Hash offset = 0 Hash preprocess = 0 Ecmp Resilient Hash = Disabled ECMP DLB Load Balancing Options: --------------------------------------------------- Load Balancing Method : Disabled Inactivity Interval : 0 (us) Flowset Table size : 0 (entries per ECMP) Reassignment Probability Threshold : 0 Reassignment Quality Delta : 0 Egress Port Load Weight : 0 EgressBytes Min Threshold : 1 EgressBytes Max Threshold : 100 Sampling Rate : 62500 Ether Type : All
Disable Queue-Pair Hashing
To exclude the queue pair from the load balancing hash calculation, configure the following:
Platform Support
See Feature Explorer for platform and release support.