Understanding Load Balancing in an FCoE-FC Gateway Proxy Fabric

You can balance the load on the ports in an FCoE-FC gateway proxy fabric in order to avoid overutilizing or underutilizing the links. The switch uses different load-balancing methods for the FC fabric-facing NP_Port links and for the FCoE-facing Ethernet links.

The switch performs automatic load balancing for the NP_Port links (the native FC interfaces) between the gateway and the FC switch using either of two algorithms:

• Simple load balancing—Each ENode fabric login (FLOGI) session and VN_Port fabric discovery (FDISC) session is assigned to the least-loaded link. This is the default load-balancing algorithm.
• ENode-based load balancing—When an ENode logs in to the fabric, all subsequent VN_Port FDISC sessions associated with that ENode are placed on the same link as the ENode FLOGI session, regardless of the link load. New ENode FLOGIs are placed on the least-loaded link.

Note: A session is a FLOGI or FDISC login to the FC SAN fabric. Session does not refer to end-to-end server-to-storage sessions.

Note: Changing the load-balancing algorithm when FCoE sessions are running forces the FCoE sessions to log out and then log in again.

You can balance the load on the Ethernet ports facing the FCoE devices by configuring those ports as a link aggregation group (LAG).

Load Balancing on the Fibre Channel Interfaces (NP_Port Links)

To balance the load on the FC fabric-facing links, you can use the default simple load-balancing algorithm or you can configure a gateway fabric to use ENode-based load balancing.

Simple load balancing provides the most equal load balancing across links because each VN_Port FDISC session can be assigned to the least-loaded link, regardless of whether the parent ENode FLOGI session is on that link. (The parent ENode is the ENode that originates the logins to the fabric. After the parent ENode logs in, the VN_Ports on that ENode can log in to the fabric using FDISC.)

ENode-based load balancing can result in a less balanced load across the NP_Port links because the VN_Port FDISC sessions are assigned to the same link as the parent ENode FLOGI session, regardless of how many FDISC sessions are associated with the ENode. However, ENode-based load balancing has the advantage of keeping all of the sessions associated with a particular ENode on one link, which provides better control and predictability.

Note: If an NP_Port link goes down, the ENode and VN_Port sessions on that link are logged out. The ENodes and VN_Port sessions log in again and are assigned to NP_Port links based on the link load and the load-balancing algorithm. If a link goes down, the switch does not rebalance the remaining load on the remaining links to avoid disrupting the existing ENode and VN_Port sessions. When you use the simple load-balancing algorithm, an ENode and its associated VN_Port sessions might be on different links. In that case, if the NP_Port with the ENode goes down, only the VN_Ports on the same link are logged out. VN_Ports on other links remain up and running.

The maximum number of FIP login sessions configured for each NP_Port interface affects load balancing. When an interface reaches its maximum number of FIP login sessions, that interface is removed from the list of interfaces used for load balancing. The other interfaces in the gateway fabric continue to accept ENode login sessions until they reach their configured maximum session limit. Only interfaces that have not reached their maximum session limit are included in the load-balancing calculations.

Note: If all NP_Port interfaces in a gateway fabric reach their FIP login session limits, the fabric sends subsequent multicast discovery advertisements (MDAs) with the availability bit set to 0 (zero) to prevent additional ENode login attempts. While the maximum number of sessions is running on the gateway fabric, ENodes cannot use that fabric to log in to the FC switch. When the number of sessions falls below the maximum, the gateway sets the availability bit in MDAs to 1 so that ENodes can log in to the fabric again.

This section describes:

• Simple Load Balancing
• ENode-Based Load Balancing
• Interface Load Algorithm
• Simple Interface Load-Balancing Scenario
• ENode-Based Interface Load-Balancing Scenarios

Simple Load Balancing

The FCoE-FC gateway performs simple load balancing by default on the NP_Ports that connect the gateway to the FC SAN. When an ENode sends a FLOGI request to the gateway, the gateway checks the NP_Ports that connect it to the FC SAN and assigns the new session to the least-loaded interface.

Every time an ENode sends a FLOGI or an FDISC request, the gateway assigns the new session to the least-loaded NP_Port. After the gateway assigns a session to an NP_Port for an ENode’s original FLOGI, subsequent FDISC requests by the same ENode can result in sessions on different NP_Ports, because the gateway always assigns the new session to the least-loaded interface.

Note: Because VN_Port sessions might be placed on a different link than their parent ENode, if the link that contains the ENode goes down, only the ENode session and any of its VN_Port sessions that are on that link go down. VN_Port sessions on other links remain active as long as the link is up and the VN_Port is not logged out.

ENode-Based Load Balancing

When you use the ENode-based load-balancing algorithm, the gateway assigns the ENode to an NP_Port link when the ENode sends its FLOGI message to the gateway. The gateway places the ENode session on the least-loaded link at that time. The VN_Port FDISC sessions associated with an ENode are placed on the same link as the ENode FLOGI session, regardless of the link load. Essentially, the ENode sessions are load-balanced, but the VN_Port sessions are not.

ENode-based load balancing ensures that each ENode and its associated VN_Port sessions are assigned to the same NP_Port link. ENode-based load balancing provides more control and predictability and ensures that if the link carrying an ENode goes down, all of the ENode’s associated VN_Port sessions also go down.

The disadvantage of ENode-based load balancing is that if one ENode has a large number of sessions and the other ENodes do not, the link that carries the ENode with the large number of sessions might have a much larger load than the other NP_Port links in the gateway proxy fabric.

For example, if a gateway fabric has two NP_Ports connected to the FC fabric, and two ENodes log in to the fabric, one ENode session is placed on each link. If two VN_Port sessions are initiated on one of the ENodes, those sessions are placed on the same link as the parent ENode. If 1000 VN_Port sessions are initiated on the other ENode, all of the 1000 VN_Port sessions are placed on the same link as that ENode. In this case, one link has 3 sessions (1 ENode FLOGI session and 2 VN_Port FDISC sessions) and the other link has 1001 sessions (1 ENode FLOGI session and 1000 VN_Port FDISC sessions).

Interface Load Algorithm

A weighted round-robin (WRR) algorithm determines the interface load based on:

• The current number of sessions on the interface
• The interface weight, which is the speed of the FC link (2 Gbps, 4 Gbps, or 8 Gbps)

The interface load algorithm is:

(number-of-sessions * max-weight) / weight

where max-weight is an internal constant.

If the load on the FC interfaces is equal, the session is assigned to the interface with the highest link speed (the greatest weight).

For example, if the three FC interfaces have the characteristics shown in Table 1, the loads of the interfaces are not equal:

In this example, interfaces fc-0/0/0 and fc-0/0/2 have a greater load than fc-0/0/1. For simple load balancing, the gateway assigns the next new FLOGI or FDISC to fc-0/0/1 because it is the least-loaded interface. For ENode-based load balancing, the gateway assigns the next new FLOGI to fc-0/0/1 because it is the least-loaded interface. Then all VN_Port FDISCs from that ENode follow the ENode and are also assigned to fc-0/0/1 regardless of the link load.

For another example, if the three FC interfaces have the characteristics shown in Table 2, the loads of the interfaces are equal:

In this case, all interfaces have the same relative load. For simple load balancing, the gateway assigns the next new FLOGI or FDISC to fc-0/0/2 because although the loads of the three interfaces are equal, fc-0/0/2 has the greatest weight. For ENode-based load balancing, the gateway assigns the next new FLOGI to fc-0/0/2, and all VN_Port FDISCs from that ENode follow the ENode and are also assigned to fc-0/0/2 regardless of the link load.

After the gateway establishes a session between an ENode or a VN_Port and an FC switch on an NP_Port, the session remains on that NP_Port until the ENode or VN_Port performs a LOGO.

If the physical FC interface link goes down, the FLOGI and FDISC sessions on the down link are logged out, and the ENodes and VN_Ports log in again to start new sessions on other NP_Ports (assuming there is more than one NP_Port connected to the FC fabric).

Simple Interface Load-Balancing Scenario

Simple load balancing results in the most equal load distribution among the NP_Ports connected to an FC SAN fabric because VN_Port FDISC sessions do not need to “follow” the parent ENode FLOGI session on the same link between the gateway and the FC fabric. When a new FLOGI or FDISC session is initiated, it is assigned to the least-loaded link.

The example in this section uses the topology shown in Figure 1.

Figure 1: Sample Load-Balancing Topology

• QFX Series configured as an FCoE-FC gateway
• Two gateway NP_Ports, fc-0/0/0 and fc-0/0/1, connected to an FC SAN fabric switch at a speed of 8 Gbps
• Three ENodes, ENode_A, ENode_B, and ENode_C connected to the gateway
• NP_Ports fc-0/0/0 and fc-0/0/1, and ENode_A, ENode_B, and ENode_C, belong to the same local FC fabric on the gateway

1. ENode_A sends a FLOGI to log in to the fabric. Because the loads on the two NP_Ports are equal, the session for ENode_A is randomly placed on one of the links. In this example, the ENode_A FLOGI session is on port fc-0/0/0.
2. Enode_B logs in. Because the load is less on port fc-0/0/1, the Enode_B FLOGI session is placed on port fc-0/0/1.
3. ENode_C logs in. Because the link loads are equal, the ENode_C login session is randomly placed on one of the links. In this example, the ENode_C login session is on port fc-0/0/0.
4. A VN_Port on ENode_A sends an FDISC to log in to the fabric. Because port fc-0/0/1 currently is the least-loaded link, the VN_Port session is placed on port fc-0/0/1, even though its parent ENode session is on port fc-0/0/0.
5. As each new VN_Port session comes up, it is placed on the least-loaded link, regardless of the link on which its parent ENode session is placed.

ENode-Based Interface Load-Balancing Scenarios

ENode-based load balancing ensures that VN_Port FDISC sessions are placed on the same link as their parent ENode FLOGI sessions, regardless of the link load. ENode-based load balancing can result in a less-balanced load among the NP_Port links, but it provides the control and predictability of keeping ENodes and their VN_Port sessions on the same link.

The examples in this section use the topology shown in Figure 1.

• QFX Series configured as an FCoE-FC gateway
• Two gateway NP_Ports, fc-0/0/0 and fc-0/0/1, connected to an FC SAN fabric switch at a speed of 8 Gbps
• Three ENodes connected to the gateway:
  • ENode_A, which will have 2 VN_Port FDISC sessions
  • ENode_B, which will have 20 VN_Port FDISC sessions
  • ENode_C, which will have 100 VN_Port FDISC sessions
• NP_Ports fc-0/0/0 and fc-0/0/1, and ENode_A, ENode_B, and ENode_C, belong to the same local FC fabric on the gateway

When the NP_Ports initialize, they send FLOGI messages to the FC switch and log in to the FC SAN fabric. The gateway then advertises the fabric to the ENodes on the Ethernet side of the network. At this point, the load on both of the NP_Ports is equal. Now the ENodes and VN_Ports start to log in to the fabric. As the following two scenarios show, how these sessions are placed on the links depends on the sequence in which they log in to the fabric.

1. ENode_A sends a FLOGI to log in to the fabric. Because the loads on the two NP_Ports are equal, the session for ENode_A is randomly placed on one of the links. In this example, the ENode_A FLOGI session is on port fc-0/0/0.
2. ENode_B logs in. Because the load is less on port fc-0/0/1, the ENode_B FLOGI session is placed on port fc-0/0/1.
3. The two VN_Ports on ENode_A log in to the fabric. Their sessions are placed on port fc-0/0/0, following ENode_A on the link. Now port fc-0/0/0 has a greater load (one FLOGI session plus two FDISC sessions) than port fc-0/0/1 (one FLOGI session).
4. The 20 VN_Ports on ENode_B log in to the fabric. Their sessions are placed on port fc-0/0/1, following ENode_B on the link. Now port fc-0/0/0 has a lesser load (one FLOGI, two FDISC) than port fc-0/0/1.
5. ENode_C logs in. Because the load is less on port fc-0/0/0, the ENode_C FLOGI session is placed on port fc-0/0/0.
6. The 100 VN_Ports on ENode_C log in to the fabric. Their sessions follow the ENode_C session onto port fc-0/0/0.
7. If more VN_Ports come up, their FDISC sessions are placed on the same link as the corresponding parent ENode session.

1. ENode_A sends a FLOGI to log in to the fabric. Because the loads on the two NP_Ports are equal, the session for ENode_A is randomly placed on one of the links. In this example, the ENode_A FLOGI session is on port fc-0/0/0.
2. ENode_B logs in. Because the load is less on port fc-0/0/1, the ENode_B FLOGI session is placed on port fc-0/0/1.
3. The two VN_Ports on ENode_A log in to the fabric. Their sessions are placed on port fc-0/0/0, following ENode_A on the link. Now port fc-0/0/0 has a greater load (one FLOGI session plus two FDISC sessions) than port fc-0/0/1 (one FLOGI session).
4. Here the login sequence in Scenario 2 differs from the login sequence in Scenario 1, resulting in a different placement of sessions on the links, and therefore a different load on the links. ENode_C logs in. Because the load is less on port fc-0/0/1, the ENode_C FLOGI session is placed on port fc-0/0/1.
5. The 20 VN_Ports on ENode_B log in to the fabric. Their sessions are placed on port fc-0/0/1, following ENode_B on the link. Now port fc-0/0/0 carries one FLOGI and two FDISC sessions, and port fc-0/0/1 carries two FLOGI and 20 FDISC sessions.
6. The 100 VN_Ports on ENode_C log in to the fabric. Their sessions follow the ENode_C session onto port fc-0/0/1. Now port fc-0/0/1 carries 2 FLOGI and 120 FDISC sessions, whereas port fc-0/0/0 carries one FLOGI and two FDISC sessions.
7. If more VN_Ports come up, their FDISC sessions are placed on the same link as the corresponding parent ENode session.

Because of the sequence of ENode logins in Scenario 2, port fc-0/0/1 carries a greater load than port fc-0/0/0. If the simple load-balancing algorithm had been used, the FLOGI and FDISC sessions would be allocated evenly to the two links. However, because the FDISC sessions are placed on the same link as their parent FLOGI sessions, using the ENode-based load-balancing algorithm can lead to scenarios in which the link loads are not equal.

Load Balancing on the FCoE Interfaces (Ethernet Links)

You can achieve load balancing for the 10-Gigabit Ethernet interfaces that belong to an FCoE VLAN by configuring them in one or more link aggregation groups (LAGs). In addition to load balancing, LAGs offer the advantages of protecting against port failover, increasing available link bandwidth, and preventing spanning-tree algorithms from blocking physical links and wasting bandwidth.