ON THIS PAGE
Example: Configuring an FCoE LAG on a Redundant Server Node Group
This example shows how to configure a Fibre Channel over Ethernet (FCoE) link aggregation group (LAG) on a redundant server Node group (RSNG) to transport FCoE traffic and regular Ethernet traffic across the same link aggregation bundle. The FCoE servers have converged network adapters (CNAs) and communicate with the Fibre Channel (FC) storage area network (SAN). FCoE servers are usually connected to a switch that performs FIP snooping, such as an FCoE transit switch or an FCoE-FC gateway switch that performs FIP snooping. This example provides a common FCoE LAG configuration for an FCoE transit switch and an FCoE-FC gateway, and shows how to disable FIP snooping scaling on an FCoE untrusted FCoE-FC gateway fabric (fc-fabric).
Requirements
This example uses the following hardware and software components:
Two Juniper Networks QFabric System Node devices configured as an RSNG. The Node devices can be configured as FCoE transit switches or as FCoE-FC gateways. (A configuration with one Node device as an FCoE transit switch and the other Node device as an FCoE-FC gateway is possible providing that the transit switch and the FCoE-FC gateway use different FCoE VLANs.)
Junos OS Release 13.2X52-D10 or later for the QFX Series
One FCoE server with two CNA ports
Overview
Standard LAGs use a hashing algorithm to determine which physical link in the LAG is used for a transmission, so a series of communications between two devices might use different physical links in the LAG for different transmissions. However, FCoE traffic requires a point-to-point link (or a virtual point-to-point link) between the FCoE device and the Fibre Channel (FC) storage area network (SAN) switch.
An FCoE LAG solves this problem by ensuring that the same LAG link is used for communication between a given FCoE device and the QFabric system Node device, preserving point-to-point link emulation. At the same time, regular Ethernet traffic (traffic that is not FCoE traffic) on the LAG is distributed across member interfaces in the same way as on a standard LAG. FCoE traffic is treated properly in terms of maintaining a virtual point-to-point link with the FC SAN, and regular Ethernet traffic enjoys the usual LAG benefits of load balancing and link redundancy.
Configuring a LAG as an FCoE LAG does not provide link redundancy for FCoE traffic, and does not load balance FCoE traffic.
On FCoE-FC gateway untrusted Fibre Channel fabrics (fc-fabrics), if you configure an FCoE LAG, you must also disable enhanced FIP snooping scaling (scaling up to 2,500 sessions), which reduces the number of supported FIP snooping sessions to 376 sessions. On an FCoE-FC gateway, disabling enhanced FIP snooping scaling is global to the Node device. Trusted fc-fabrics on an FCoE-FC gateway support enhanced FIP snooping scaling.
This example shows you how to:
Configure the RSNG and its Node devices
Configure the FCoE LAG on the RSNG
Configure a dedicated VLAN for FCoE traffic (an FCoE VLAN) and a native VLAN for untagged FCoE initialization protocol (FIP) traffic
Enable VN2VF_Port FIP snooping on the FCoE VLAN
Disable FIP snooping scaling on an untrusted FCoE-FC gateway fabric
FCoE traffic requires lossless transport across the Ethernet network to comply with the requirements for transporting storage traffic. This example describes how to configure an FCoE LAG to provide redundancy for FCoE traffic. See Example: Configuring CoS PFC for FCoE Traffic for how to configure lossless transport for FCoE traffic.
On a Node device that is configured as an FCoE-FC gateway, you must create a Fibre Channel fabric, configure native FC interfaces, configure an FCoE VLAN interface (a Layer 3 RVI) for the FCoE VLAN (which includes the FCoE LAG as a member interface), and add the native FC interfaces and FCoE VLAN interface to the FC fabric. For an example of FCoE-FC gateway interface configuration, see Example: Setting Up Fibre Channel and FCoE VLAN Interfaces in an FCoE-FC Gateway Fabric.
Topology
Table 1 shows the configuration components for this example.
Component |
Settings |
|---|---|
Hardware |
Two QFabric system Node devices configured as an RSNG (the Node devices can be configured as FCoE transit switches or as FCoE-FC gateways; this example is valid for both modes):
Note:
The alias names chosen for this example indicate the physical locations of the Node devices. You can use any aliasing system you want to make identifying Node devices easier, or you can use the default Node device names (the Node device serial numbers). One FCoE server with two CNA ports. |
LAG configuration |
RSNG device count—48 FCoE LAG name—RSNG1:ae20 FCoE LAG member interfaces—row1rack1:xe-0/0/20 and row1rack2:xe-0/0/20 FCoE LAG LACP—active FCoE LAG port mode—trunk MTU—2180 FCoE LAG VLAN memberships—FCoE VLAN ( |
FCoE VLAN |
Name— VLAN ID— Member interfaces— |
Native VLAN |
Name— VLAN ID— Member interfaces— |
VN2VF_Port FIP snooping |
Enabled on the FCoE VLAN ( |
FIP snooping scaling |
Enabled for FCoE transit switch portion of the example. Disabled for the FCoE-FC gateway portion of the example (gateway FC fabric is FCoE untrusted). |
Figure 1 shows the network topology for this example.
Configuration
To configure an FCoE LAG between an FCoE server with two CNA ports and the two Node device members of an RSNG, perform these tasks:
- CLI Quick Configuration
- Configuring an FCoE LAG on an RSNG (FCoE Transit Switch or FCoE-FC Gateway)
- Disabling Enhanced FIP Snooping Scaling on an FCoE-FC Gateway
- Results
CLI Quick Configuration
In this example, the enhanced FIP snooping scaling is disabled (376 sessions) on the FCoE-FC gateway because the gateway fabric is an untrusted fc-fabric.
Most of the FCoE LAG configuration is common to both the FCoE transit switch and FCoE-FC gateway modes of operation. The CLI Quick Configuration shows the common configuration statements first, followed by the additional configuration statement that disables FIP snooping scaling on the FCoE-FC gateway. Disabling FIP snooping scaling on an FCoE-FC gateway is a global configuration that affects all of the fc-fabrics on the gateway. (On an FCoE transit switch, you can disable FIP snooping scaling on an individual FCoE VLAN without affecting other FCoE VLANs.)
This example does not include configuring the FC fabric, the native FC fabric ports, and the Layer 3 FCoE VLAN interface.
To quickly configure an FCoE LAG , copy the following commands, paste them in a text file, remove line breaks, change variables and details to match your network configuration, and then copy and paste the commands into the CLI at the [edit] hierarchy level.
Common configuration:
set fabric aliases node-device ABCD1234 row1-rack1 set fabric aliases node-device ABCD1235 row1-rack2 set fabric resources node-group RSNG1 node-device row1-rack1 set fabric resources node-group RSNG1 node-device row1-rack2 set chassis node-group RSNG1 aggregated-devices ethernet device-count 48 set interfaces RSNG1:ae20 unit 0 family ethernet-switching port-mode trunk vlan members fcoe-vlan1 set interfaces RSNG1:ae20 unit 0 family ethernet-switching native-vlan-id 1 set interfaces RSNG1:ae20 mtu 2180 set interfaces RSNG1:ae20 aggregated-ether-options fcoe-lag set interfaces RSNG1:ae20 aggregated-ether-options lacp active set interfaces row1-rack1:xe-0/0/20 ether-options 802.3ad RSNG1:ae20 set interfaces row1-rack2:xe-0/0/20 ether-options 802.3ad RSNG1:ae20 set vlans fcoe-vlan1 vlan-id 2000 set vlans native vlan-id 1 set vlans fcoe-vlan1 interface RSNG1:ae20 set ethernet-switching-options secure-access-port vlan fcoe-vlan1 examine-fip
If you want to configure an FCoE-FC gateway fabric as a trusted fabric so that you can leave enhanced FIP snooping scaling enabled on the gateway, add the following statement to the configuration, replacing the variable fc-fabric-name with the name of the FC fabric (if you do this, do not disable FIP snooping scaling as shown in the FCoE-FC Gateway Additional Configuration):
set fc-fabrics fc-fabric-name protocols fip fcoe-trusted
Additional configuration to disable enhanced FIP snooping scaling on an FCoE-FC gateway untrusted FC fabric:
set fc-options no-fip-snooping-scaling
Configuring an FCoE LAG on an RSNG (FCoE Transit Switch or FCoE-FC Gateway)
Step-by-Step Procedure
To configure the RSNG member Node devices, the FCoE LAG, the FCoE VLAN, and VN2VF_Port FIP snooping on an FCoE transit switch or an FCoE-FC gateway:
Define aliases for the two Node devices that will be in the RSNG (aliases are easier to remember and more descriptive than the Node device serial number). Name the Node device with serial number
ABCD1234asrow1-rack1and the Node device with the serial numberABCD1235asrow1-rack2:admin@qfabric# set fabric aliases node-device ABCD1234 row1-rack1 admin@qfabric# set fabric aliases node-device ABCD1235 row1-rack2
Configure the Node device membership for
row1-rack1androw1-rack2in the RSNGRSNG1:admin@qfabric# set fabric resources node-group RSNG1 node-device row1-rack1 admin@qfabric# set fabric resources node-group RSNG1 node-device row1-rack2
Configure the number of LAG interfaces that RSNG
RSNG1can support. (Each Node device in the RSNG has 48 server-facing ports. If we used one port from each Node device to provide Node device redundancy for each LAG, we might need to support a maximum of 48 LAGs, so we set the device count to48LAGs.)admin@qfabric# set chassis node-group RSNG1 aggregated-devices ethernet device-count 48
Configure the LAG interface (
ae20) on RSNG1 and set the port mode totrunkmode. In the same statement, configure the LAG interface membership in the dedicated FCoE VLANfcoe-vlan1:admin@qfabric# set interfaces RSNG1:ae20 unit 0 family ethernet-switching port-mode trunk vlan members fcoe-vlan1
Configure the LAG interface membership in the native VLAN:
admin@qfabric# set interfaces RSNG1:ae20 unit 0 family ethernet-switching native-vlan-id 1
Configure the LAG interface with an MTU of
2180to accommodate the size of the FCoE frame and headers.admin@qfabric# set interfaces RSNG1:ae20 mtu 2180
Configure the LAG
RSNG1:ae20as an FCoE LAG:admin@qfabric# set interfaces RSNG1:ae20 aggregated-ether-options fcoe-lag
Enable LACP on the FCoE LAG:
admin@qfabric# set interfaces RSNG1:ae20 aggregated-ether-options lacp active
Assign one Ethernet interface on each RSNG Node device to the FCoE LAG:
admin@qfabric# set interfaces row1-rack1:xe-0/0/20 ether-options 802.3ad RSNG1:ae20 admin@qfabric# set interfaces row1-rack2:xe-0/0/20 ether-options 802.3ad RSNG1:ae20
Configure a dedicated VLAN for FCoE traffic (an FCoE VLAN) named
fcoe-vlan1with the VLAN ID2000:admin@qfabric# set vlans fcoe-vlan1 vlan-id 2000
Configure a native VLAN with the VLAN ID
1to carry untagged FIP traffic:admin@qfabric# set vlans native vlan-id 1
Assign the FCoE LAG interface to the FCoE VLAN:
admin@qfabric# set vlans fcoe-vlan1 interface RSNG:ae20
Assign the FCoE LAG interface to the native VLAN:
admin@qfabric# set vlans native interface RSNG:ae20
Enable VN2VF_Port FIP snooping on the FCoE VLAN:
admin@qfabric# set ethernet-switching-options secure-access-port vlan fcoe-vlan1 examine-fip
Disabling Enhanced FIP Snooping Scaling on an FCoE-FC Gateway
Step-by-Step Procedure
To disable enhanced FIP snooping scaling on an FCoE-FC gateway:
Disable FIP snooping scaling on the gateway fabrics. Disabling FIP snooping scaling on an FCoE-FC gateway is global to the gateway, so every FC fabric on the gateway reverts to supporting 376 sessions (instead of 2,500 sessions as with FIP snooping scaling enabled).
admin@qfabric# set fc-options no-fip-snooping-scaling
Results
Display the results of the configuration. The results below show the configuration on an FCoE transit switch and have been edited to include only the components configured in the example:
admin@qfabric> show configuration
root@qfabric>fabric {
resources {
node-group RSNG1 {
node-device row1-rack1;
node-device row1-rack2;
}
}
aliases {
node-device ABCD1234 {
row1-rack1;
}
node-device ABCD1235 {
row1-rack2;
}
}
}
chassis {
node-group RSNG1 {
aggregated-devices {
ethernet {
device-count 48;
}
}
}
}
interfaces {
RSNG1:ae20 {
aggregated-ether-options {
fcoe-lag;
lacp {
active;
}
}
unit 0 {
family ethernet-switching {
port-mode trunk;
vlan {
members fcoe-vlan1;
}
native-vlan-id 1;
}
}
}
row1-rack1:xe-0/0/20 {
ether-options {
802.3ad RSNG1:ae20;
}
}
row1-rack2:xe-0/0/20 {
ether-options {
802.3ad RSNG1:ae20;
}
}
}
ethernet-switching-options {
secure-access-port {
vlan fcoe-vlan1 {
examine-fip;
}
}
}
vlans {
fcoe-vlan1 {
vlan-id 2000;
interface {
RSNG1:ae20.0;
}
}
native {
vlan-id 1;
interface {
RSNG1:ae20.0;
}
}
}
Verification
To verify the configuration of the QFabric system Node device resources, FCoE LAG, FCoE VLAN, native VLAN, and FIP snooping, perform these tasks:
- Verifying the Node Device Aliases (Names)
- Verifying the Node device Assignment to the Node Group
- Verifying the Number of Aggregated Ethernet Logical Devices (LAG Interfaces) That the Node Group Can Support
- Verifying the FCoE LAG Interface Configuration
- Verifying the FCoE VLAN and Native VLAN Configuration
- Verifying the FIP Snooping Configuration
Verifying the Node Device Aliases (Names)
Purpose
Verify that the Node device alias names are configured.
Action
List the Node device inventory on the QFabric system
using the show fabric administration inventory node-devices command:
admin@qfabric> show fabric administration inventory node-devices root@qfabric>show fabric administration inventory node-devices Item Identifier Connection Model Node device row1-rack1 ABCD1234 Connected qfx3500 row1-rack2 ABCD1235 Connected qfx3500
Meaning
The show fabric administration inventory node-devices command lists the Node device names in the Node device column and lists the Node device serial numbers in the Identifier column. The Connection column shows if the Director device has detected the Node device,
and the Model column lists QFX switch model type.
The command output shows that Node device ABCD1234 is configured with the name (alias) row1-rack1, and the
Node device ABCD1235 is configured with the name row1-rack2.
Verifying the Node device Assignment to the Node Group
Purpose
Verify that the redundant server Node group includes the two Node devices.
Action
Verify that the QFabric system Node group RSNG1 is configured with the correct Node devices using the show
configuration fabric resources command:
admin@qfabric> show configuration fabric resources
root@qfabric> show configuration fabric resources
node-group RSNG1 {
node-device row1-rack1;
node-device row1-rack2;
}
Meaning
The show configuration fabric resources command
lists the Node groups and the Node devices in the Node groups. The
command output shows that Node group RSNG1 consists of
the Node devices row1-rack1 and row1-rack2.
Verifying the Number of Aggregated Ethernet Logical Devices (LAG Interfaces) That the Node Group Can Support
Purpose
Verify the number of LAG interfaces that the redundant server node group supports.
Action
List the LAG interface device count using the show
configuration chassis command:
admin@qfabric> show configuration chassis
node-group RSNG1 {
aggregated-devices {
ethernet {
device-count 48;
}
}
}Meaning
The show configuration chassis command displays
the Ethernet device count (the number of LAG interfaces supported)
as 48 devices.
Verifying the FCoE LAG Interface Configuration
Purpose
Verify that the FCoE LAG interface, port mode, interface VLAN membership, and Node device interface membership in the FCoE LAG are correctly configured.
Action
List the FCoE LAG interface and Node device interface
information using the show configuration interfaces command:
admin@qfabric> show configuration interfaces
RSNG1:ae20 {
aggregated-ether-options {
fcoe-lag;
lacp {
active;
}
}
unit 0 {
family ethernet-switching {
port-mode trunk;
vlan {
members fcoe-vlan1;
}
native-vlan-id 1;
}
}
}
row1-rack1:xe-0/0/20 {
ether-options {
802.3ad RSNG1:ae20;
}
}
row1-rack2:xe-0/0/20 {
ether-options {
802.3ad RSNG1:ae20;
}
}
Meaning
The show configuration interfaces command
lists both the LAG interfaces and the individual Node device interfaces,
and their configuration.
The command output shows a lot of information about the interfaces:
The LAG interface name is
RSNG1:ae20fcoe-lagconfirms the LAG is an FCoE LAGlacpis configured inactivemodePort mode is
trunkThe LAG has membership in the
fcoe-vlan1VLAN and in the native VLAN with the VLAN ID1.Interface
row1-rack1:xe-0/0/20is a member of FCoE LAGRSNG1:ae20Interface
row1-rack2:xe-0/0/20is a member of FCoE LAGRSNG1:ae20
Verifying the FCoE VLAN and Native VLAN Configuration
Purpose
Verify that the FCoE VLAN fcoe-vlan1 and
the native VLAN native are configured with the correct
VLAN tags (2000 and 1, respectively) and that
the FCoE LAG interface RSNG1:ae20 is assigned to the VLANs.
Action
List the VLAN information using the show configuration
vlans command:
admin@qfabric> show configuration vlans
fcoe-vlan1 {
vlan-id 2000;
interface {
RSNG1:ae20.0;
}
}
native {
vlan-id 1;
interface {
RSNG1:ae20.0;
}
Meaning
The show configuration vlans command lists
the configured VLANs, their VLAN IDs, and the interfaces assigned
to the VLANs.
The command output shows that the FCoE VLAN fcoe-vlan1 is configured with the VLAN ID 2000 and is assigned to
the FCoE LAG interface RSNG1:ae20.
The command output also shows that the native VLAN native is configured with the VLAN ID 1 and is assigned to the
FCoE LAG interface RSNG1:ae20.
Verifying the FIP Snooping Configuration
Purpose
Verify that VN2VF_Port FIP snooping is enabled on the
FCoE VLAN (fcoe-vlan1).
Action
List the FIP snooping information using the show
configuration ethernet-switching-options command:
admin@qfabric> show configuration ethernet-switching-options
secure-access-port {
vlan fcoe-vlan1 {
examine-fip;
}
}
Meaning
The show configuration ethernet-switching-options command lists the security options configured on VLANs. The command
output shows that on VLAN fcoe-vlan1, VN2VF_Port FIP snooping
is enabled (examine-fip output).