Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

 
 

AI Storage for AI-ML Data Centers

AI and machine learning workloads require a high-performance, lossless Ethernet fabric to transport large volumes of data efficiently between servers, storage systems, and AI processing devices such as GPUs. Storage plays a critical role in supporting data-intensive operations such as model training, inference, checkpointing, and dataset access. Junos OS Evolved provides storage-focused networking capabilities that help ensure predictable throughput, low latency, and efficient utilization of network resources in AI-ML data centers.

AI storage architectures generally fall into two categories:

  • NVMe-oF/TCP storage— Enables hosts to access remote NVMe SSDs over an Ethernet network and provides high-performance block storage. It allows storage resources to be disaggregated from compute resources and shared across multiple servers. This architecture is commonly used in AI and machine learning environments for training datasets, model checkpoints, and other data-intensive workloads, where efficient access to flash storage and scalable storage pools are critical for maximizing infrastructure utilization and application performance.

  • Parallel file system storage— Distributes data across multiple storage nodes and provides a shared file system that can be accessed simultaneously by many compute and accelerator nodes. These architectures are designed to deliver scalability, parallelism, and aggregate bandwidth required to support large-scale AI training and inference workloads.

Table 1: Key Characteristics of NVMe-oF/TCP and Parallel File System Storage
Characteristic NVMe-oF/TCP Parallel File System
Storage model Block storage Shared file storage
Access method Direct access to remote NVMe devices Shared file system namespace
Primary goal Low-latency access to flash storage High-throughput shared data access
Typical scale Individual hosts or storage clients Large AI clusters
Common use cases Databases, checkpoints, storage pools AI training datasets, model development, data lakes

AI Storage Networking

AI storage deployments typically consist of two distinct network domains:

  • Front-end storage network— Carries storage-access traffic between compute nodes, GPUs, and storage services. This network is optimized for low latency, congestion mitigation, and predictable performance to maximize accelerator utilization.
  • Back-end storage network— Carries storage-internal traffic between storage nodes, including data replication, erasure coding, synchronization, rebuild operations, and metadata exchanges. This network is optimized for high bandwidth, scalability, and efficient traffic forwarding.

This section describes storage-specific networking features supported on AI-ML data center platforms and how they can be used to optimize storage traffic in AI deployments.

DCBX with ETS for AI-ML Data Centers

Data Center Bridging Exchange (DCBX) is a key protocol used in modern data center networks to support enhanced Ethernet capabilities in converged networking environments. DCBX operates as an LLDP extension that discovers, exchanges, and negotiates Data Center Bridging (DCB) features such as Priority-based Flow Control (PFC) and Enhanced Transmission Selection (ETS). On QFX Series switches, ETS can be advertised and discovered over LLDP through the use of DCBX ETS Configuration (Subtype 9) and Recommendation (Subtype A) Type-Length-Value (TLV) entries. This enables network devices to discover traffic class mappings, priority group assignments, bandwidth allocations, and transmission selection settings, helping ensure consistent QoS policies and reduce configuration errors across the data center network.

In AI-ML data centers, multiple traffic types, including storage, management, and application traffic, often share the same Ethernet infrastructure. To ensure predictable performance in these environments, network operators can use Enhanced Transmission Selection (ETS), configured through Data Center Bridging Exchange (DCBX), to allocate bandwidth among traffic classes according to application requirements.

ETS is particularly useful in front-end storage networks, which connects compute and accelerator nodes to shared storage systems. In these environments, ETS helps ensure that storage traffic receives its configured share of bandwidth while allowing compute and accelerator traffic to share the remaining network resources according to defined bandwidth policies.

  • Automatically shares traffic and bandwidth policies.
  • Ensures consistent QoS across the network.
  • Reduces manual configuration and errors.
  • Helps optimize AI-ML and RoCE workload performance through consistent bandwidth allocation.
  • Supports efficient GPU communications and simplifies network operations.

Consider an example AI cluster where bandwidth is allocated as follows:

  • 60% of bandwidth is allocated to RoCE traffic used for GPU synchronization.
  • 25% is allocated to storage traffic.
  • 10% is allocated to inference traffic.
  • 5% is allocated to management traffic.

With ETS advertisement and discovery, neighboring switches and servers can learn these bandwidth allocations and verify that traffic classes are aligned. This helps ensure that distributed training workloads receive the bandwidth they require while minimizing the impact of storage and management traffic on GPU communications.

Table 2: DCBX with ETS
Topic Use Case
DCBX and ETS Learn the fundamentals of DCBX ETS advertisement, including default ETS advertisement behavior, ETS advertisement and peer configuration, and the use of ETS Recommendation TLVs.