Appendix A – How to Run a NCCL Test Using Autoconfigured IPv6 Address
To run a model or NCCL test using a global IPv6 addresses assigned either statically or automatically via SLAAC the value of the NCCL_IB_GID_INDEX variable must be adjusted. The NCCL_IB_GID_INDEX variable defines the Global ID index used by RoCE (RDMA) communication.
The default value is -1, which means that NCCL will automatically select the correct GID index based on the active link layer of the InfiniBand device. If the link layer is Ethernet (RoCE), NCCL will use the GID index that returns a GID with RoCE v2 support (usually GID index 3, depending on driver/firmware).
For more details you can review Nvidia’s Environment Variables — NCCL 2.27.3 documentation
To find the GID for the desired address, use the following command:
ibv_devinfo -vvv -d <mellanox-interface-name> | grep GID
To find the mellanox interface name you can use the following script:
jnpr@H100-01:~/scripts$ cat nvidia_map_iface_to_mlx_YL.sh
# Script to map network interfaces to Mellanox interfaces
echo "Network Interface to Mellanox Interface Mapping:"
# Loop through each network interface in /sys/class/net/
for iface in $(ls /sys/class/net/); do
if [ -d /sys/class/net/$iface/device/infiniband_verbs ]; then
# Find the Mellanox interface by reading the ibdev file
mlx_iface=$(cat /sys/class/net/$iface/device/infiniband_verbs/*/ibdev)
echo "$iface => $mlx_iface"
fi
doneExample:
jnpr@H100-01:/etc/netplan$ ibv_devinfo -vvv -d mlx5_6 | grep GID GID[ 0]: fe80:0000:0000:0000:a288:c2ff:fe3b:506a, RoCE v1 GID[ 1]: fe80::a288:c2ff:fe3b:506a, RoCE v2 GID[ 2]: 0000:0000:0000:0000:0000:ffff:0ac8:010a, RoCE v1 GID[ 3]: ::ffff:10.200.1.10, RoCE v2 GID[ 4]: 2010:0200:0000:0002:a288:c2ff:fe3b:506a, RoCE v1 GID[ 5]: 2010:200:0:2:a288:c2ff:fe3b:506a, RoCE v2 jnpr@H100-01:~/scripts$ ./nvidia_map_iface_to_mlx_YL.sh | egrep "gpu|Map" Network Interface to Mellanox Interface Mapping: gpu0_eth => mlx5_11 gpu1_eth => mlx5_6 gpu2_eth => mlx5_10 gpu3_eth => mlx5_9 gpu4_eth => mlx5_4 gpu5_eth => mlx5_3 gpu6_eth => mlx5_5 gpu7_eth => mlx5_0 gpu6_eth => mlx5_5 gpu7_eth => mlx5_0 stor0_eth => mlx5_1
Once you have identified the GID you can run a NCCL test as shown in the example:
user@headend-svr-1:~$ NCCL_PXN_DISABLE=1 NCCL_IB_QPS_PER_CONNECTION=4 NCCL_IB_GID_INDEX =5 ./nccl_run_rails_all_H100.sh -b 1G -e 1G -n 200 -i 0 -m 10
The following script provides mapping information between the Mellanox interface names, the NIC numbers, and the user assigned interface names (e.g. gpu0_eth). It also provides mapping information between the interfaces and the GPUs.
jnpr@A100-01:~/SCRIPTS$ cat find_pxb_gpu_nic_pairs.py
#!/usr/bin/env python3
import subprocess
import pandas as pd
import re
from collections import defaultdict
# Step 1: Run filter_topo_dynamic.py
print("🚀 Running filter_topo_dynamic.py...")
subprocess.run(["python3", "filter_topo_dynamic.py"], check=True)
# Step 2: Read filtered_topo.csv
df = pd.read_csv("filtered_topo.csv")
# Step 3: Identify PXB entries from GPU rows
pxb_matches = defaultdict(list)
gpu_rows = df[df["Label"].str.startswith("GPU")]
for _, row in gpu_rows.iterrows():
gpu = row["Label"]
for nic in df.columns[1:]:
if str(row[nic]).strip().upper() == "PXB":
pxb_matches[gpu].append(nic)
# Step 4: Parse gpu_eth-to-nic.txt
gpu_eth_to_nic = {}
with open("gpu_eth-to-nic.txt") as f:
for line in f:
match = re.match(r"(gpu\d+_eth)\s+←→\s+(NIC\d+)", line)
if match:
gpu_eth, nic = match.groups()
gpu_eth_to_nic[gpu_eth] = nic
# Step 5: Invert map to find which GPU eth corresponds to each NIC
nic_to_gpu_eth = {nic: gpu_eth for gpu_eth, nic in gpu_eth_to_nic.items()}
# Step 6: Output result
print("\n🎯 GPU to PXB NICs (with eth):")
for gpu in sorted(pxb_matches.keys(), key=lambda x: int(x[3:])):
nic_list = sorted(pxb_matches[gpu], key=lambda x: int(x[3:]))
formatted = ", ".join([f"{nic} ({nic_to_gpu_eth.get(nic, 'unknown')})" for nic in nic_list])
print(f"{gpu} => {formatted}")
Example:
jnpr@A100-01:~/SCRIPTS$ python3 find_pxb_gpu_nic_pairs.py Running filter_topo_dynamic.py... [sudo] password for jnpr: Mapping from mlx5_X to gpuX_eth: mlx5_6 → gpu0_eth mlx5_8 → gpu1_eth mlx5_0 → gpu2_eth mlx5_2 → gpu3_eth mlx5_16 → gpu4_eth mlx5_18 → gpu5_eth mlx5_10 → gpu7_eth mlx5_12 → gpu6_eth Saved to mlx_to_gpu_eth.txt NIC Legend: mlx5_0 → NIC0 mlx5_1 → NIC1 mlx5_2 → NIC2 mlx5_3 → NIC3 mlx5_4 → NIC4 mlx5_5 → NIC5 mlx5_6 → NIC6 mlx5_7 → NIC7 mlx5_8 → NIC8 mlx5_9 → NIC9 mlx5_10 → NIC10 mlx5_11 → NIC11 mlx5_12 → NIC12 mlx5_13 → NIC13 mlx5_14 → NIC14 mlx5_15 → NIC15 mlx5_16 → NIC16 mlx5_17 → NIC17 mlx5_18 → NIC18 mlx5_19 → NIC19 Matched NICs and their GPUs: gpu0_eth ←→ NIC6 gpu1_eth ←→ NIC8 gpu2_eth ←→ NIC0 gpu3_eth ←→ NIC2 gpu4_eth ←→ NIC16 gpu5_eth ←→ NIC18 gpu7_eth ←→ NIC10 gpu6_eth ←→ NIC12 Saved to gpu_eth-to-nic.txt Done! Filtered output saved to filtered_topo.csv GPU to PXB NICs (with eth): GPU0 => NIC6 (gpu0_eth), NIC8 (gpu1_eth) GPU1 => NIC6 (gpu0_eth), NIC8 (gpu1_eth) GPU2 => NIC0 (gpu2_eth), NIC2 (gpu3_eth) GPU3 => NIC0 (gpu2_eth), NIC2 (gpu3_eth) GPU4 => NIC16 (gpu4_eth), NIC18 (gpu5_eth) GPU5 => NIC16 (gpu4_eth), NIC18 (gpu5_eth) GPU6 => NIC10 (gpu7_eth), NIC12 (gpu6_eth) GPU7 => NIC10 (gpu7_eth), NIC12 (gpu6_eth)