CoS on Virtual Chassis Fabric (VCF) EX4300 Leaf Devices (Mixed Mode)
A Virtual Chassis Fabric (VCF) uses QFX5100 switches as spine devices and can use QFX5100, QFX3500, QFX3600, and EX4300 switches as leaf devices. When a VCF includes more than one type of leaf device (mixed mode), the CoS feature support on the VCF depends on the capability of the lowest-featured device. In mixed mode, the supported CoS features are the “lowest common denominator” of the features supported by the leaf devices. If one leaf device does not support a particular feature, that feature is not supported on the VCF even if every other leaf device supports the feature.
EX4300 leaf devices do not support several CoS features that are supported on QFX5100, QFX3600, and QFX3500 devices. However, even when a VCF includes an EX4300 leaf device, other leaf devices might support those CoS features.
VCF CoS in Mixed Mode with an EX4300 Leaf Device
In mixed mode, if all of the leaf devices are QFX5100, QFX3500, and QFX3600 switches, the full QFX Series CoS feature set is available, including data center bridging (DCB) features such as enhanced transmission selection (ETS, IEEE 802.1Qaz), priority-based flow control (PFC, IEEE 802.1Qbb), and Data Center Bridging Exchange Protocol (DCBX, an extension of LLDP, IEEE 802.1AB).
However, the EX4300 leaf device does not support DCB standards (ETS, PFC, DCBX). The lack of support for DCB standards means that the EX4300 leaf device does not support lossless transport. So a VCF that includes an EX4300 as a leaf device does not support lossless storage traffic such as Fibre Channel over Ethernet (FCoE).
In addition, a VCF with an EX4300 leaf device either does not support or has limited support for some other CoS features that the QFX Series switches support, including some buffer configuration features, some packet rewrite features, and Ethernet PAUSE (IEEE 802.3X).
Table 1 summarizes the CoS support on a VCF in mixed mode with one or more EX4300 leaf devices.
QFX Series CoS Feature |
Support in Mixed Mode with an EX4300 Leaf Device |
---|---|
Forwarding Classes |
The EX4300 leaf device uses the QFX Series default forwarding classes, the default QFX Series forwarding class to queue mapping, and the QFX Series maximum number of supported forwarding classes (12). |
Lossless Forwarding Classes |
Not supported. For example, the QFX Series default lossless forwarding classes |
Shared buffer configuration |
Ingress shared buffer configuration is not supported. Egress shared buffer configuration does not support partitioning into three buffer pools. If there is a shared buffer configuration, only the total egress shared buffer configuration is used. Ingress shared buffer configuration and egress buffer partitioning configuration is ignored. |
Classifier on a Layer 2 interface |
One classifier per protocol is supported on a port. On a physical port, for a particular protocol, the same Layer 2 classifier is used on all of the logical interfaces. |
Classifier on a Layer 3 interface |
Supported. |
Multi-destination classifier |
Supported. The EX4300 leaf device uses the same default classifier as the QFX5100 spine device. As on QFX Series switches, a multi-destination classifier is global and is applied to all VCF interfaces. Multi-destination classifiers are valid only for multicast forwarding classes. You can configure two multi-destination classifiers, one for IEEE 802.1p traffic and one for DSCP traffic (the DSCP multi-destination classifier applies to both IPv4 and IPv6 traffic). |
Congestion notification profile |
Not supported. If a congestion notification profile is configured on the QFX5100 spine device, it is ignored because the EX4300 leaf device does not support lossless transport, so end-to-end lossless behavior is not possible |
Ethernet PAUSE (IEEE 802.3X) |
Not supported. If Ethernet PAUSE is configured, it is ignored. |
Hierarchical scheduling (ETS) |
Translated into port-based scheduling. The EX4300 device does not support ETS scheduling. A VCF translates ETS scheduling configured on a QFX5100 spine device into port scheduling on an EX4300 leaf device. The hierarchical structure of mapping forwarding classes into forwarding class sets (fc-sets) is ignored. Scheduling on an EX4300 VCF Leaf Device provides details on how a VCF translates QFX Series ETS scheduling into port scheduling on an EX4300 leaf device. |
Hierarchical scheduling (ETS) on a spine device VCP port |
On QFX5100 VCP ports, the hierarchical mapping of forwarding classes to forwarding class sets is supported. However, scheduling on an EX4300 leaf device is translated into port scheduling. |
Drop profile (WRED) |
QFX Series drop profiles are supported. The EX4300 device as a standalone switch supports four packet loss priorities. However, as part of a mixed mode VCF, the EX4300 leaf device supports only the three packet loss priorities that the QFX Series switches support:
Supporting only three packet loss priorities means that the behavior of the EX4300 switch as a leaf device is different from the behavior as a standalone switch. |
Rewrite rules on a Layer 2 interface |
Supported, but with a limit of one rewrite rule per physical interface. All traffic uses the same rewrite rule. |
Rewrite rules on a Layer 3 interface |
Supported, but with a limit of one rewrite rule per physical interface. The same rewrite rule is used on all traffic on the interface. |
Rewrite value for FCoE traffic |
Not supported. If a rewrite value for FCoE traffic, is configured, it is ignored. (A mixed mode VCF does not support lossless traffic.) |
In addition to the CoS limitations shown in Table 1, using wild cards in a LAG configuration is not supported in mixed mode with one or more EX4300 leaf devices.
Scheduling on an EX4300 VCF Leaf Device
Because the EX4300 leaf device does not support ETS, the VCF translates the ETS scheduling configuration into the port scheduling configuration that the EX4300 device supports. The QFX5100 spine device uses two-tier ETS scheduling, as described in detail in Understanding CoS Hierarchical Port Scheduling (ETS).
Briefly, ETS allocates port bandwidth into forwarding class sets (priority groups) and forwarding classes (priorities) in a hierarchical manner. Each forwarding class set consists of individual forwarding classes, with each forwarding class mapped to an output queue.
Port bandwidth (minimum guaranteed bandwidth and maximum bandwidth) is allocated to each forwarding class set. Forwarding class set bandwidth is in turn allocated to the forwarding classes in the forwarding class set. If a forwarding class does not use its bandwidth allocation, other forwarding classes within the same forwarding class set can share the unused bandwidth. If the forwarding classes in a forwarding class set do not use the bandwidth allocated to that forwarding class set, other forwarding class sets on the port can share the unused bandwidth. (This is how ETS increases port bandwidth utilization, by sharing unused bandwidth among forwarding classes and forwarding class sets.)
However, the EX4300 leaf device supports port scheduling, not ETS. Port scheduling is a “flat” scheduling method that allocates bandwidth directly to forwarding classes in a non-hierarchical manner.
The VCF translates the two tiers of the ETS scheduling configuration (forwarding class sets and forwarding classes) into a single port scheduling configuration as follows:
The bandwidth allocated to a forwarding class set is divided equally among the forwarding classes in the forwarding class set. (Traffic control profiles schedule bandwidth allocation to forwarding class sets.) The minimum guaranteed bandwidth (
guaranteed-rate
) and maximum bandwidth limit (shaping-rate
) of the forwarding class set determine the guaranteed minimum bandwidth and the maximum bandwidth the forwarding classes receive, unless those values are different in the forwarding class scheduler configuration.If there is an explicit forwarding class bandwidth scheduler configuration, it overrides the forwarding class set configuration. Bandwidth scheduling values that are not explicitly configured in a forwarding class scheduler use the values from the forwarding class set (the traffic control profile configuration). Forwarding class schedulers control the minimum guaranteed bandwidth (
transmit-rate
), the maximum bandwidth (shaping-rate
), and the priority (priority
) for each forwarding class (output queue). Because the priority value is not configured at the forwarding class set level, the priority configured in the forwarding class scheduler is always used.
The following two scenarios illustrate how a VCF translates an ETS configuration into a port scheduling configuration:
Scenario 1
A forwarding class set named fc-set-1
has a configured
guaranteed minimum bandwidth (guaranteed-rate
) of 4G, and
a configured maximum bandwidth (shaping-rate
) of 5G.
Forwarding class set fc-set-1
consists of two forwarding
classes, named fc-1
and fc-2
:
Forwarding class
fc-1
has a guaranteed minimum bandwidth (transmit-rate
) of 2.5G. There is no configured maximum bandwidth (shaping-rate
).Forwarding class
fc-2
has a guaranteed minimum bandwidth (transmit-rate
) of 1.5G. There is no configured maximum bandwidth (shaping-rate
).
On the EX4300 leaf device, the ETS configuration above is translated approximately to the following port scheduling configuration:
Guaranteed minimum bandwidth—Because guaranteed minimum bandwidth has been explicitly configured in the forwarding class scheduler, forwarding class
fc-1
receives a transmit rate of 2.5G and forwarding classfc-2
receives a transmit rate of 1.5G.Note:If there had been no forwarding class scheduler
transmit-rate
configuration, then the forwarding class set minimum guaranteed bandwidth of 4G would have been split evenly between the forwarding classes, with each forwarding class receiving a minimum guaranteed bandwidth rate of 2G.Maximum bandwidth—Because there is no explicit maximum bandwidth (
shaping-rate
configuration for the forwarding classes, the forwarding classes that belong to the forwarding class set receive an equal share of the maximum bandwidth configured at the forwarding class set level in the traffic control profile. Because the forwarding class set maximum bandwidth is 5G, forwarding classesfc-1
andfc-2
each receive a maximum bandwidth of 2.5G.
In this scenario, the minimum guaranteed bandwidth and the maximum
bandwidth configured at the forwarding class set hierarchy level are
achieved on the forwarding classes that belong to the forwarding class
set. (This does not always happen, as Scenario 2 shows.) However,
unused bandwidth is not shared the same way. For example, if forwarding
class fc-1
experienced a burst of traffic at 3.5G, it would
be limited to a maximum of 2.5G and traffic would be dropped. Using
ETS, if forwarding class fc-2
was not using its allocated
maximum bandwidth, then fc-1
could use (share) that unused
bandwidth. But flat port scheduling does not share the unused bandwidth.
Scenario 2
A forwarding class set named fc-set-2
has a configured
guaranteed minimum bandwidth (guaranteed-rate
) of 6G, and
a configured maximum bandwidth (shaping-rate
) of 9G.
Forwarding class set fc-set-2
consists of three forwarding
classes, named fc-3
, fc-4
, and fc-5
:
Forwarding class
fc-3
has a guaranteed minimum bandwidth (transmit-rate
) of 1G. There is no configured maximum bandwidth (shaping-rate
).Forwarding class
fc-4
has a maximum bandwidth (shaping-rate
) of 2G. There is no configured guaranteed minimum bandwidth (transmit-rate
).Forwarding class
fc-5
has a guaranteed minimum bandwidth (transmit-rate
) of 3G. There is no configured maximum bandwidth (shaping-rate
).
On the EX4300 leaf device, the ETS configuration above is translated approximately to the following port scheduling configuration:
Guaranteed minimum bandwidth—Two forwarding classes (
fc-3
andfc-5
) have an explicitly configured transmit rate, and one forwarding class (fc-4
) does not. Forwarding classesfc-3
andfc-5
receive the minimum guaranteed bandwidth configured in their schedulers, so forwarding classfc-3
receives 1G guaranteed minimum bandwidth and forwarding classfc-5
receives 3G guaranteed minimum bandwidth.Forwarding class
fc-4
does not have an explicitly configured transmit rate, so the port derives the minimum guaranteed bandwidth from the forwarding class set guaranteed rate. Forwarding class setfc-set-2
has a minimum guaranteed bandwidth (guaranteed-rate
) of 6G, and there are three forwarding classes in the forwarding class set. Forwarding classfc-4
receives an equal share (one third) of the forwarding class set minimum guaranteed bandwidth. So forwarding classfc-4
is allocated a guaranteed minimum bandwidth (transmit-rate
) of 2G (6G divided by 3 forwarding classes = 2G).Maximum bandwidth—Forwarding class
fc-4
has an explicitly configured shaping rate, and forwarding classesfc-3
andfc-5
do not. Forwarding classfc-4
receives the maximum bandwidth configured in its scheduler, so forwarding classfc-4
receives a maximum bandwidth of 2G.Forwarding classes
fc-3
andfc-5
do not have explicitly configured shaping rates, so the port derives the maximum bandwidth from the forwarding class set shaping rate. Forwarding class setfc-set-2
has a maximum bandwidth (shaping-rate
) of 9G, and there are three forwarding classes in the forwarding class set. Forwarding classesfc-3
andfc-5
each receive an equal share (one third) of the forwarding class set shaping rate. So forwarding classesfc-3
andfc-5
are allocated a maximum bandwidth of 3G each (9G divided by 3 forwarding classes = 3G).Forwarding class
fc-4
receives less maximum bandwidth than forwarding classesfc-3
andfc-5
because the explicitly configured shaping rate for forwarding classfc-4
is only 2G, and the explicit forwarding class configuration overrides the forwarding class set configuration.
Scenario 2 shows that in some cases, the guaranteed minimum
bandwidth (guaranteed-rate
) and the maximum bandwidth (shaping-rate
) configured for a forwarding class set might not
be achieved at the forwarding class (queue) level. In Scenario 2,
forwarding class set fc-set-2
has a shaping rate of 9G,
but the sum of the implemented forwarding class shaping rates is only
8G [(3G for fc-3
) + (2G for fc-4
) + (3G for fc-5
)].