Multi-node (Load Balanced) Inference

In multi-node load balanced inference, GenAI-Perf sends inference requests to an Envoy endpoint rather than directly to an MI300X inference server. Envoy then distributes requests across available MI300X inference servers running model serving endpoints. After a request reaches an inference server, the SGLang Router distributes it to local GPU-backed workers, as described in the single-node inference flow.

This mode represents a production-style inference deployment model where users or applications access a single frontend service endpoint while inference capacity is distributed across multiple servers.

In the example shown in Figure 4, GenAI-Perf runs on Lambda Scaler 2 using IP address 10.10.1.34 and sends traffic to Envoy running on Lambda Scaler 1 using IP address 10.10.1.28. The Envoy frontend endpoint listens on destination port 8000. After receiving a request, Envoy forwards the request to one of the MI300X inference servers.

The diagram shows Envoy forwarding traffic to two inference servers:

MI300-01 using destination address 10.10.5.25 and destination port 30000.

MI300-02 using destination address 10.10.6.27 and destination port 30000.

In this mode, the source address seen by the MI300X inference servers is the Envoy host. This allows the benchmark client to target a single frontend endpoint while Envoy performs request distribution across the inference servers.

Table 8: Multi-node (Load Balanced) Inference Summary

Field	Example / Purpose
Source	GenAI-Perf client host
Example source address	10.10.1.34
Frontend endpoint	Envoy load balancer
Example Envoy address	10.10.1.28
Envoy frontend port	8000
Backend endpoints	MI300X inference servers running SGLang
Example server 1	MI300-01, 10.10.15.25, destination port 30000
Example server 2	MI300-02, 10.10.6.27, destination port 30000
Traffic behavior	GenAI-Perf sends requests to Envoy. Envoy forwards requests to inference servers.
Purpose	Scale-out inference validation across multiple inference servers.