Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

Understanding FCoE LAGs

 

A Fibre Channel over Ethernet (FCoE) link aggregation group (LAG) is a special LAG that enables you to transport FCoE traffic and regular Ethernet traffic across the same link aggregation bundle. Fibre Channel (FC) storage area network (SAN) switches require a point-to-point connection (or a virtual point-to-point connection) to FCoE devices. This requirement means that communication between an FCoE device and a QFabric system Node device must use the same physical link in a LAG to maintain the virtual point-to-point connection.

However, a standard LAG can use any member link for any particular transmission, so a standard LAG cannot guarantee that the same link is used for requests and responses between an FCoE device and a QFabric system Node device. Using different LAG member links for communication breaks the virtual point-to-point connection, which results in dropped FCoE traffic.

Unlike a standard LAG, an FCoE LAG always uses the same member link to transmit FCoE traffic between an FCoE device and a QFabric system Node device. However, regular Ethernet traffic (traffic is that is not FCoE traffic) on the LAG is distributed across member interfaces in the same way as on a standard LAG. The special treatment of FCoE traffic does not affect the way regular Ethernet traffic behaves on the LAG. FCoE traffic is treated properly in terms of maintaining a virtual point-to-point link, and regular Ethernet traffic enjoys the usual LAG benefits of load balancing and link redundancy.

Note

Configuring a LAG as an FCoE LAG does not provide link redundancy for FCoE traffic, and does not load balance FCoE traffic.

A LAG interface can be a member of both an FCoE VLAN and a regular Ethernet VLAN. An FCoE LAG allows FCoE and standard Ethernet traffic to coexist on the same LAG, and treats both types of traffic properly.

On QFabric systems, all of the member links of an FCoE LAG must belong to one Node group. The member links of an FCoE LAG cannot belong to different Node groups.

Like a standard LAG, an FCoE LAG can have up to 32 member interfaces. FCoE devices are usually servers with CNAs connected to a switch that performs FIP snooping, such as an FCoE transit switch or an FCoE-FC gateway switch that performs FIP snooping.

Why a Standard LAG Does Not Work for FCoE Traffic

Each physical link that carries FCoE traffic connects to a CNA port on an FCoE device. The connection that the FIP process creates between the CNA and the FC SAN switch emulates a point-to-point connection between that CNA and the SAN switch through the QFabric system Node device. If a connection to an FCoE device is not on a point-to-point link, communication from the FC SAN switch to the FCoE device CNA might not reach the CNA.

In a LAG, two (or more) physical links connect to the same device. Standard LAGs use a hashing algorithm to determine which physical LAG link to use for each transmission. Because the hashing algorithm might choose any LAG link for a given transmission, there is no way a standard LAG can guarantee that a response from the FC SAN will use the same LAG link on a Node device as the request from the CNA.

To ensure that communication between the CNA and the FC SAN is successful, communication from the SAN to the CNA must use the same physical link. If the FCoE CNA sends a request to the FC SAN, the response from the FC SAN must come on the same link the FCoE device CNA used to send the request. For example, if a if a request from the CNA goes out on Node device LAG member interface RSNG1:xe-0/0/20, then the response from the FC SAN must be received on interface RSNG1:xe-0/0/20.

If the FC SAN switch response to the FCoE CNA uses a different physical link on the Node device LAG, the response arrives at a different CNA port than the CNA port on which the request was sent. This breaks the virtual point-to-point link and the SAN switch response does not reach the correct requestor, so the response is lost. This is why a standard LAG does not work for FCoE traffic.

How an FCoE LAG Works

For FIP and FCoE transactions with the FC SAN to work properly, a LAG for FIP and FCoE traffic must allow the FC SAN switch to respond to the FCoE CNA device on the same link that the CNA used to communicate with the FC SAN switch.

To accomplish this, an FCoE LAG selects the member interface that the CNA used to communicate with the FC SAN switch as the link for the SAN switch response to the CNA. This preserves the virtual point-to-point link across the LAG and ensures that traffic from the FC SAN reaches the correct CNA port.

In a standard LAG, other devices learn the MAC address of the LAG interface, not the MAC address of the physical member interface that actually carries the traffic. However, for FCoE communication, other devices need to learn and use the VN_Port MAC address that the SAN switch assigns to the virtual node port (VN_Port) on the FCoE device’s CNA. The VN_Port MAC address uniquely identifies the CNA port used for FCoE transmission. (The VN_Port MAC address is based on the Fibre Channel ID and the FC-MAP value, which the FC SAN switch provides to the FCoE CNA as a unique port identifier.)

In an FCoE LAG, the Node device performs FIP snooping to learn the VN_Port MAC address of the CNA (in addition to other information). The Node device assigns the VN_Port MAC address to the particular interface that was used to connect to the CNA. For FCoE traffic, this replaces the normal LAG hashing logic, so instead of using an arbitrary LAG interface on the Node device for FCoE communication between the SAN switch and the CNA, an FCoE LAG uses the same physical LAG link for all FCoE transactions based on the VN_Port MAC address.

VLAN discovery traffic is untagged, so it must use a native VLAN. When you configure an FCoE LAG, VLAN discovery traffic on a native VLAN in the LAG also automatically uses the same physical link, preserving the virtual point-to-point link.

For multicast packets such as multicast discovery advertisements (MDAs), the advertisement is forwarded on all member links of the FCoE LAG. This ensures that multicast advertisements reach all of the FCoE devices attached to FCoE LAG member interfaces.

Behavior on FCoE LAG Link Failure

If an FCoE LAG link goes down, FCoE traffic and regular Ethernet traffic are treated differently.

If an FCoE LAG link goes down, the FCoE sessions on that link also go down. The Node device cannot simply move a session to another LAG link because that breaks the virtual point-to-point link. FCoE LAGs do not provide link redundancy for FCoE traffic.

As on a normal LAG, an FCoE LAG provides link redundancy for regular Ethernet traffic. Regular Ethernet sessions on the down FCoE LAG link are moved to other member links of the FCoE LAG (assuming that other member links are up).

FIP Snooping Session Scaling on QFabric System Node Devices

When the switch is on the FCoE access edge, you must enable FIP snooping on the FCoE VLAN to provide secure access when connecting to the FC SAN. (You can also enable FIP snooping on FCoE VLANs on switches that are not at the access edge if you want to collect FIP snooping statistics on the switch or if you are not confident that the edge switch is properly snooping traffic.)

FIP snooping VLANs support scaling up to 2,500 sessions by default, which is called enhanced FIP snooping scaling mode. Software releases before Junos OS Release 12.3 limited VN2VF_Port FIP snooping session scaling to 376 sessions on untrusted interfaces and untrusted FC fabrics, but scaled to 2,500 sessions on trusted interfaces and trusted FC fabrics. Starting with Junos OS Release 12.3, by default, all VN2VF_Port FIP snooping VLANs used enhanced FIP snooping scaling (2,500 sessions) for both trusted and untrusted interfaces and FC fabrics. The old limit of 376 sessions for untrusted interfaces and untrusted FC fabrics was deprecated and could not be configured.

The FCoE LAG feature introduces the ability to disable FIP snooping session scaling so that only 376 sessions are supported instead of the default 2,500 sessions. The reason for reintroducing FIP snooping session scaling limits is that when a Node device is configured as an FCoE-FC gateway that has one or more untrusted gateway Fibre Channel fabric (fc-fabric), placing FCoE traffic in a LAG forces the TCAM to store additional session data to ensure that the virtual point-to-point link between the FCoE device and the FC SAN is maintained. This case is described later in this document.

FCoE LAG Configuration on an FCoE Transit Switch

To create an FCoE LAG on an FCoE transit switch, you include the fcoe-lag option in the [edit interfaces interface-name aggregated-ether-options] hierarchy.

In addition to creating the FCoE LAG, you also need to:

  • Add interfaces to the FCoE LAG.

  • Configure at least one dedicated VLAN for FCoE traffic (an FCoE VLAN).

  • Configure a native VLAN to carry untagged FIP traffic.

  • Configure the FCoE LAG interfaces as a member of both the FCoE VLAN and the native VLAN.

  • Enable FIP snooping on the FCoE VLAN.

FCoE LAG Configuration and FIP Snooping Scaling on an FCoE-FC Gateway

There are differences in the way you configure an FCoE LAG on an FCoE-FC gateway compared to configuring an FCoE LAG on an FCoE transit switch.

Configuring an FCoE LAG on an FCoE-FC Gateway

To create an FCoE LAG on an FCoE-FC gateway, you include the fcoe-lag option in the [edit interfaces interface-name aggregated-ether-options] hierarchy.

In addition to creating the FCoE LAG, you also need to:

  • Add interfaces to the FCoE LAG.

  • Configure at least one dedicated VLAN for FCoE traffic (an FCoE VLAN).

  • Configure a native VLAN to carry untagged FIP traffic.

  • Configure the FCoE LAG interfaces as a member of both the FCoE VLAN and the native VLAN.

  • Configure an FCoE VLAN interface (a Layer 3 routed VLAN interface that is configured as a virtual F_Port) for the FCoE traffic. This enables the FCoE VLAN (and the member FCoE LAG interfaces) to interface with the with the native Fibre Channel ports in the FCoE-FC gateway switch Fibre Channel fabric (fc-fabric).

  • Add the FCoE VLAN interface to the fc-fabric.

  • Enable FIP snooping on the FCoE VLAN.

  • Configure FIP snooping session scaling as described in the next section. The FIP snooping scaling mode depends on whether the fc-fabric is trusted or untrusted.

FIP Snooping Session Scaling on an FCoE-FC Gateway

FIP snooping session scaling on an FCoE-FC gateway depends on whether or not the gateway has an untrusted fc-fabric:

  • If the FCoE-FC gateway fc-fabric is FCoE trusted, then you can use enhanced FIP snooping scaling (2,500 sessions), and you do not have to do any additional configuration even if two or more FCFs in an FCoE VLAN have the same FC-MAP value.

  • If the FCoE-FC gateway fc-fabric is FCoE untrusted, then you must disable enhanced FIP snooping scaling (reduce the number of supported sessions to 376 sessions) by including the no-fip-snooping-scaling statement in the [edit fc-options] hierarchy.

    Note

    On an FCoE-FC gateway, disabling enhanced FIP snooping scaling is global.

Gateway fc-fabrics are untrusted by default. FCoE-FC gateways do not support FCoE LAGs on untrusted fc-fabrics when enhanced FIP snooping scaling is enabled.

Summary of FCoE LAG and FIP Snooping Scaling on an FCoE-FC Gateway

Table 1 summarizes FCoE LAG and FIP snooping scaling on an FCoE-FC gateway.

Table 1: Summary of FCoE LAG and FIP Snooping Scaling (FCoE-FC Gateway)

FCoE Fabric Trusted or Untrusted

FCoE LAG Configured

FIP Snooping Session Scaling

Configuration Notes

Trusted

Yes (fcoe-lag option included in the [edit interfaces interface-name aggregated-ether-options] hierarchy)

2,500 sessions (enhanced FIP snooping scaling)

Configure the fc-fabric as an FCoE trusted fabric by including the fcoe-trusted option in the [edit fc-fabrics fc-fabric-name protocols fip fcoe-trusted] hierarchy.

Untrusted

Yes (fcoe-lag option included in the [edit interfaces interface-name aggregated-ether-options] hierarchy)

376 sessions (no FIP snooping scaling)

Disable FIP snooping scaling by including the no-fip-snooping-scaling option in the [edit fc-options] hierarchy. This disables FIP snooping scaling globally on the gateway.

Untrusted

No (fcoe-lag option not included in LAG configuration)

2,500 sessions (enhanced FIP snooping scaling)

FCoE LAGs with enhanced FIP snooping scaling enabled are not supported on untrusted FCoE-FC gateway fc-fabrics.

To configure an FCoE LAG on an untrusted fc-fabric, FIP snooping scaling must be disabled.

FCoE Blade Switches

If you are using an FCoE blade switch, you need to configure an FCoE LAG only if the blade switch uses a passthrough module instead of an integrated switch.

Limitations

There are several limitations to configuring FCoE LAGs:

  1. All FCoE LAG member links must belong to the same QFabric system Node group.

  2. On an FCoE-FC gateway, you must disable FIP snooping scaling on untrusted fc-fabrics. Disabling FIP snooping scaling is global to the gateway Node device. If all of the fc-fabrics on an FCoE-FC gateway are trusted fabrics, you do not need to disable FIP snooping scaling.

  3. FCoE LAGs with enhanced FIP snooping scaling enabled are not supported on untrusted FCoE-FC gateway fc-fabrics.