Benchmarking Testing Methodology

The validated benchmark methodology uses NVIDIA GenAI-Perf to generate high volumes of inference for each of the models described in the Validated Models section.

Both single-node and multi-node load balanced scenarios as described in the Validated Inference flows.

All the test scenarios were completed following these steps:

1. Generates benchmark inference requests from the client host using GenAI-Perf either directly to an MI300X inference server or to the Envoy Load Balancer.

   When generating benchmark inference requests following parameters were adjusted to extract the best performance, and the following inference metrics were collected:

   Table 11: GenAI-Perf Inference request parameters.

   Parameter	Description
   Concurrency	Number of simultaneous inference requests or active request streams generated during the benchmark test. Increasing concurrency helps evaluate how the inference service behaves under higher demand.
   Number of Requests	Total number of inference requests generated during the benchmark run. This value determines the size of the test sample used to calculate performance metrics.
   Input Sequence Length (ISL) [tokens]	Number of input tokens included in each request or prompt sent to the inference service.
   Output Sequence Length (OSL) [tokens]	Number of output tokens generated by the model in each response.
   Warm-up	Number of initial requests sent before the measured benchmark run begins. These requests are not included in the final reported performance results. Warm-up helps reduce the effect of startup behavior, model initialization, caching, and other transient conditions.

   ISL represents the prompt size, while OSL represents the generated response size. In general, larger ISL values increase the amount of input the model must process before generating the first token, while larger OSL values increase the amount of generated output produced during the response.

2. Collect and analyze inference performance metrics generated by GenAI-Perf

The following inference performance metrics were included:

Table 12. Inference Performance Metrics

Catagory	Metric	Description
Responsiveness	Time to First Token (TTFT)	Measured in milliseconds. Elapsed time between request submission and the first generated token. Indicates perceived responsiveness and interactive user experience.
Responsiveness	Time to First Output (TTFO)	Measured in milliseconds. Elapsed time between request submission and the first output received by the client. For streaming text generation, this is often closely related to TTFT.
Request Latency	Request Latency	Measured in milliseconds. Total elapsed time from request submission to completed response. Captures full user-facing response time.
Streaming Performance	Time to Second Token (TTST)	Measured in milliseconds. Elapsed time between request submission and the second generated token. Helps characterize early streaming behavior after the first token is returned.
Streaming Performance	Inter-Token Latency (ITL)	Measured in milliseconds. Delay between generated output tokens during response streaming. Indicates response smoothness and token generation consistency.
Throughput	Output Tokens per Second (TPS)	Measured in tokens per second. Rate of output token generation by the system. Measures aggregate inference throughput and serving efficiency.
Throughput	Output Tokens per Second per User	Measured in tokens per second per user. Average output token generation rate experienced by each concurrent user or request stream.
Throughput	Request Throughput	Measured in requests per second. Number of inference requests completed by the system per second. Indicates how many requests the service can handle under benchmark load.

The test results can be found AI Data Center Frontend Fabric for Inference with HPE Juniper QFX switches, Apstra Data Center Director, and AMD Instinct MI300X GPUs -Juniper Validated Design (JVD) Test Report