附录 A – 如何使用自动配置的 IPv6 地址运行 NCCL 测试
要使用静态或通过 SLAAC 自动分配的全局 IPv6 地址运行模型或 NCCL 测试，必须调整 NCCL_IB_GID_INDEX 变量的值。NCCL_IB_GID_INDEX 变量定义 RoCE （RDMA） 通信使用的全局 ID 索引。
默认值为 -1，这意味着 NCCL 将根据 InfiniBand 设备的活动链路层自动选择正确的 GID 索引。如果链路层是以太网 （RoCE），NCCL 将使用返回支持 RoCE v2 的 GID（通常为 GID 索引 3，具体取决于驱动程序/固件）的 GID 索引。
有关更多详细信息，您可以查看 Nvidia 的 环境变量 — NCCL 2.27.3 文档
要查找所需地址的 GID，请使用以下命令：
ibv_devinfo -vvv -d <mellanox-interface-name> | grep GID
要查找 mellanox 接口名称，您可以使用以下脚本：
jnpr@H100-01:~/scripts$ cat nvidia_map_iface_to_mlx_YL.sh
# Script to map network interfaces to Mellanox interfaces
echo "Network Interface to Mellanox Interface Mapping:"
# Loop through each network interface in /sys/class/net/
for iface in $(ls /sys/class/net/); do
if [ -d /sys/class/net/$iface/device/infiniband_verbs ]; then
# Find the Mellanox interface by reading the ibdev file
mlx_iface=$(cat /sys/class/net/$iface/device/infiniband_verbs/*/ibdev)
echo "$iface => $mlx_iface"
fi
done
示例：
jnpr@H100-01:/etc/netplan$ ibv_devinfo -vvv -d mlx5_6 | grep GID GID[ 0]: fe80:0000:0000:0000:a288:c2ff:fe3b:506a, RoCE v1 GID[ 1]: fe80::a288:c2ff:fe3b:506a, RoCE v2 GID[ 2]: 0000:0000:0000:0000:0000:ffff:0ac8:010a, RoCE v1 GID[ 3]: ::ffff:10.200.1.10, RoCE v2 GID[ 4]: 2010:0200:0000:0002:a288:c2ff:fe3b:506a, RoCE v1 GID[ 5]: 2010:200:0:2:a288:c2ff:fe3b:506a, RoCE v2 jnpr@H100-01:~/scripts$ ./nvidia_map_iface_to_mlx_YL.sh | egrep "gpu|Map" Network Interface to Mellanox Interface Mapping: gpu0_eth => mlx5_11 gpu1_eth => mlx5_6 gpu2_eth => mlx5_10 gpu3_eth => mlx5_9 gpu4_eth => mlx5_4 gpu5_eth => mlx5_3 gpu6_eth => mlx5_5 gpu7_eth => mlx5_0 gpu6_eth => mlx5_5 gpu7_eth => mlx5_0 stor0_eth => mlx5_1
确定 GID 后，您可以运行 NCCL 测试，如示例所示：
user@headend-svr-1:~$ NCCL_PXN_DISABLE=1 NCCL_IB_QPS_PER_CONNECTION=4 NCCL_IB_GID_INDEX =5 ./nccl_run_rails_all_H100.sh -b 1G -e 1G -n 200 -i 0 -m 10
以下脚本提供 Mellanox 接口名称、NIC编号和用户分配的接口名称（例如 gpu0_eth）之间的映射信息。它还提供接口与 GPU 之间的映射信息。
jnpr@A100-01:~/SCRIPTS$ cat find_pxb_gpu_nic_pairs.py
#!/usr/bin/env python3
import subprocess
import pandas as pd
import re
from collections import defaultdict
# Step 1: Run filter_topo_dynamic.py
print("🚀 Running filter_topo_dynamic.py...")
subprocess.run(["python3", "filter_topo_dynamic.py"], check=True)
# Step 2: Read filtered_topo.csv
df = pd.read_csv("filtered_topo.csv")
# Step 3: Identify PXB entries from GPU rows
pxb_matches = defaultdict(list)
gpu_rows = df[df["Label"].str.startswith("GPU")]
for _, row in gpu_rows.iterrows():
gpu = row["Label"]
for nic in df.columns[1:]:
if str(row[nic]).strip().upper() == "PXB":
pxb_matches[gpu].append(nic)
# Step 4: Parse gpu_eth-to-nic.txt
gpu_eth_to_nic = {}
with open("gpu_eth-to-nic.txt") as f:
for line in f:
match = re.match(r"(gpu\d+_eth)\s+←→\s+(NIC\d+)", line)
if match:
gpu_eth, nic = match.groups()
gpu_eth_to_nic[gpu_eth] = nic
# Step 5: Invert map to find which GPU eth corresponds to each NIC
nic_to_gpu_eth = {nic: gpu_eth for gpu_eth, nic in gpu_eth_to_nic.items()}
# Step 6: Output result
print("\n🎯 GPU to PXB NICs (with eth):")
for gpu in sorted(pxb_matches.keys(), key=lambda x: int(x[3:])):
nic_list = sorted(pxb_matches[gpu], key=lambda x: int(x[3:]))
formatted = ", ".join([f"{nic} ({nic_to_gpu_eth.get(nic, 'unknown')})" for nic in nic_list])
print(f"{gpu} => {formatted}")
示例：
jnpr@A100-01:~/SCRIPTS$ python3 find_pxb_gpu_nic_pairs.py Running filter_topo_dynamic.py... [sudo] password for jnpr: Mapping from mlx5_X to gpuX_eth: mlx5_6 → gpu0_eth mlx5_8 → gpu1_eth mlx5_0 → gpu2_eth mlx5_2 → gpu3_eth mlx5_16 → gpu4_eth mlx5_18 → gpu5_eth mlx5_10 → gpu7_eth mlx5_12 → gpu6_eth Saved to mlx_to_gpu_eth.txt NIC Legend: mlx5_0 → NIC0 mlx5_1 → NIC1 mlx5_2 → NIC2 mlx5_3 → NIC3 mlx5_4 → NIC4 mlx5_5 → NIC5 mlx5_6 → NIC6 mlx5_7 → NIC7 mlx5_8 → NIC8 mlx5_9 → NIC9 mlx5_10 → NIC10 mlx5_11 → NIC11 mlx5_12 → NIC12 mlx5_13 → NIC13 mlx5_14 → NIC14 mlx5_15 → NIC15 mlx5_16 → NIC16 mlx5_17 → NIC17 mlx5_18 → NIC18 mlx5_19 → NIC19 Matched NICs and their GPUs: gpu0_eth ←→ NIC6 gpu1_eth ←→ NIC8 gpu2_eth ←→ NIC0 gpu3_eth ←→ NIC2 gpu4_eth ←→ NIC16 gpu5_eth ←→ NIC18 gpu7_eth ←→ NIC10 gpu6_eth ←→ NIC12 Saved to gpu_eth-to-nic.txt Done! Filtered output saved to filtered_topo.csv GPU to PXB NICs (with eth): GPU0 => NIC6 (gpu0_eth), NIC8 (gpu1_eth) GPU1 => NIC6 (gpu0_eth), NIC8 (gpu1_eth) GPU2 => NIC0 (gpu2_eth), NIC2 (gpu3_eth) GPU3 => NIC0 (gpu2_eth), NIC2 (gpu3_eth) GPU4 => NIC16 (gpu4_eth), NIC18 (gpu5_eth) GPU5 => NIC16 (gpu4_eth), NIC18 (gpu5_eth) GPU6 => NIC10 (gpu7_eth), NIC12 (gpu6_eth) GPU7 => NIC10 (gpu7_eth), NIC12 (gpu6_eth)