ON THIS PAGE
Fabric Plane Management
Configuring Fabric Redundancy Mode for Active Control Boards on MX Series Routers
An MX960 router can support three Enhanced Switch Control Boards (SCBE2s or SCBEs)—two planes on each SCB and make up a total of six fabric planes. MX240 and MX480 routers can support up to two SCBE2s or SCBEs—four fabric planes on each SCBE make up a total of eight planes. However, the MX240 and MX480 routers have only six active planes. The remaining two are redundant.
MX2020 routers can support eight Switch Fabric Boards (SFBs) or 24 fabric planes. The MX2020 chassis provides redundancy and resiliency. All major hardware components including the power system, the cooling system, the control board and the switch fabrics are fully redundant.
MX10004 supports six SFBs. Each SFB with the switch fabric is connected to the line cards and the Routing and Control Board (RCB). Three SFBs provide reduced switching functionality to an MX10004 router. Six SFBs provide full throughput. Each MX10004 SFB has four connectors. Each connector matches up with a line card slot, eliminating the need for a backplane. The MX10004 power system and the Routing Control Board (RCB) provide redundancy and resiliency.
The MX2010 and MX2020 routers support 8 SFBs and two control boards. MX2010 and MX2020 routers provide redundancy and resiliency. All major hardware components including the power system, the cooling system, the control board and the switch fabrics are fully redundant.
The MX2020 router has 20 dedicated line card slots.The host subsystem consists of two Control Boards with Routing Engines (CBREs) and eight Switch Fabric Boards (SFBs).
An MX10008 devices has six Switch Fabric Boards (SFBs). MX10K-LC2101 has six Packet Forwarding Engines (PFE). Each PFE has 24 connections to the fabric (24 planes, or 4 connections per SFB).
The MX10008 has two models of SFBs: the JNP10008-SF and the JNP10008-SF2. SFBs installed must be of the same model type in a running chassis. On both SF and SF2 models, the SFB has eight connectors that connect to one of the eight line cards.The MPC7E-MRATE and MPC7E-10G MPCs are supported only on MX-SCBE2.
You can configure the active control board to be in redundancy mode or in
increased fabric bandwidth mode. You can enable increased fabric bandwidth of active
control boards for optimal and efficient performance and traffic handling by
configuring the active control boards to be in redundancy mode. To configure
redundancy mode for the active control board, use the redundancy-mode
redundant
statement at the [edit chassis fabric]
hierarchy level:
When you configure this option, all the FPCs use 4 fabric planes as active planes, regardless of the type of the FPC.
To configure
increased bandwidth mode for the active control board, use the
redundancy-mode increased-bandwidth
statement at the
[edit chassis fabric]
hierarchy level:
In increased fabric bandwidth mode, MX Series routers will use 6 active planes. MX240 and MX480 routers will also use 2 spare planes in addition to the 6 active planes.
Increased fabric bandwidth mode is enabled by default on MX routers with Switch Control Board (SCB). On MX routers with Enhanced SCB—SCBE, regardless of the type of MPC or DPC installed on it, redundancy mode is enabled by default.
Fabric mode cannot be changed dynamically for MX-Series devices. Please reboot the MX device after switching the fabric mode.
Configuring this feature does not affect the system. You can configure this feature without restarting the FPC or restarting the system.
Example: Configuring Fabric Redundancy Mode
- Requirements for Configuration of the Fabric Redundancy Mode
- Overview
- Configuring Increased Bandwidth Mode
- Verification
Requirements for Configuration of the Fabric Redundancy Mode
This example uses the following hardware and software components:
Junos OS Release 12.3 R2 or later for MX Series routers
A single MX480 router with MPC4E
Overview
This example provides information about configuring the fabric redundancy mode on an MX480 router with MPC4E. You can configure the MPC4E to function in redundant fabric mode or increased bandwidth mode. If you do not configure the mode, the MPC4E, by default, functions in redundant fabric mode. In redundant fabric mode, the number of active fabric planes is 4. If you configure the MPC4E to function in increased bandwidth mode, the number of active fabric planes increases to 6.
See also: 32x10GE MPC4E and 2x100GE + 8x10GE MPC4E.
Configuring Increased Bandwidth Mode
Procedure
Step-by-Step Procedure
In this example, you configure increased bandwidth mode on an MX480 router with MPC4E. The existing fabric mode on the MX480 router is redundant fabric mode. To configure the fabric mode, perform the following tasks:
Verify the existing fabric mode of the router by using the
show chassis fabric mode
command.user@host > show chassis fabric mode Fabric Operating Mode : Redundant Fabric
View the number of active fabric planes by using the
show chassis fabric summary
command.user@host > show chassis fabric summary Plane State Uptime 0 Online 2 hours, 58 minutes, 22 seconds 1 Online 6 seconds 2 Online 32 seconds 3 Online 2 hours, 58 minutes, 23 seconds 4 Spare 31 seconds 5 Spare 21 seconds 6 Spare 18 seconds 7 Spare 9 seconds For FPC slots with MPC Type 4, Type 5, or MCC: Fabric planes 1 and 5, 3 and 7 use shared physical links. Those slots may run in a reduced bandwidth in case both plane 1 and 5, or both 3 and 7 are active.
Type 4 and Type 5 MPCs refer to MPC 4 and MPC5 line cards, respectively.
In configuration mode, go to the
[edit chassis]
hierarchy level and set the fabric mode toincreased-bandwidth
as follows:[edit chassis] user@ host #set fabric redundancy-mode increased-bandwidth
Results
In redundant fabric
mode, the number of
active fabric planes is 4 while the number of spare planes is also
4. In increased-bandwidth
mode, the number of active planes
is 6 while the number of spare planes is 2.
Fabric planes 1 and 5 and fabric planes 3 and 7 use shared physical links. So, among fabric planes 1 and 5, only one plane can be active. Similarly, among fabric planes 3 and 7, only one plane can be active.
Verification
To verify that the fabric mode of the MX480 router with MPC4E, perform the following tasks:
Verifying the Fabric Redundancy Mode of the Router
Purpose
To verify that the fabric redundancy mode of the MX480
router with MPC4E has been modified to increased-bandwidth
.
Action
To view the fabric mode of the router, use the show chassis fabric mode
command.
user@host > show chassis fabric mode Fabric redundancy mode: Increased Bandwidth
Meaning
The MX480 router with MPC4E is functioning in increased bandwidth mode.
Verifying the Number of Active Fabric Planes
Purpose
To verify that the number of active fabric planes is 6.
Action
To view the number of active fabric planes, use the show chassis fabric summary
command.
user@host > show chassis fabric summary Plane State Uptime 0 Online 2 hours, 55 minutes, 49 seconds 1 Online 2 hours, 55 minutes, 25 seconds 2 Online 2 hours, 58 minutes, 48 seconds 3 Online 2 hours, 55 minutes, 50 seconds 4 Online 2 hours, 55 minutes, 48 seconds 5 Spare 2 hours, 55 minutes, 40 seconds 6 Online 2 hours, 55 minutes, 37 seconds 7 Spare 2 hours, 55 minutes, 29 seconds For FPC slots with MPC Type 4, Type 5, or MCC: Fabric planes 1 and 5, 3 and 7 use shared physical links. Those slots may run in a reduced bandwidth in case both plane 1 and 5, or both 3 and 7 are active.
Type 4 and Type 5 MPCs refer to MPC 4 and MPC5 line cards, respectively.
Meaning
Number of active planes on the MX480 router with MPC4E is 6 (0, 1, 2, 3, 4, and 6) while the number of spare planes is 2.
Fabric Plane Management on AS MLC Modular Carrier Card
The Application Services Modular Line Card (AS MLC) provides high application throughput and storage space, and is designed to run services on the MX240, MX480, and MX960 routers. The AS MLC consists of the following components:
Application Services Modular Carrier Card (AS MCC)
Application Services Modular Processing Card (AS MXC)
Application Services Modular Storage Card (AS MSC)
The AS MCC plugs into the chassis and provides the fabric interface.
An MX960 router can support three Switch Control Boards (SCBs) or six fabric planes. The AS MCC supports six fabric planes. An MX240 or MX480 router can support upto two SCBs or two fabric planes. The AS MCC at any time can provide connectivity to only six of the eight fabric planes. Fabric planes 1 and 5, and 3 and 7 use shared physical links. So between fabric planes 1 and 5 only one plane can be active. Similarly between fabric planes 3 and 7, only one plane can be active.
This behavior impacts the output of fabric-related monitoring commands on MX240 and MX480 routers with AS MCCs.
The show chassis fpc pic-status
command displays
the output for an MX480 router with an AS MCC:
user@host>show chassis fpc pic-status Slot 2 Online MPC Type 1 3D Q Slot 1 Online AS-MCC PIC 0 Online AS-MSC PIC 2 Online AS-MXC Slot 4 Offline MPC 3D 16x 10GE Slot 5 Offline AS-MCC
In the show chassis fpc pic-status
command output, Slot 1 and 5 are AS MCC, PIC 0 is the AS MSC, and PIC 2 is the AS MXC.
The show chassis fabric fpcs
command displays the
output on an MX480 router with an AS MCC.
user@hostshow chassis fabric fpcs FPC 2 PFE #0 Plane 0: Plane enabled Plane 1: Plane enabled Plane 2: Plane enabled Plane 3: Plane enabled Plane 4: Plane enabled Plane 5: Plane enabled Plane 6: Plane enabled Plane 7: Plane enabled FPC 4 PFE #0 Plane 0: Plane enabled Plane 1: Plane enabled Plane 2: Plane enabled Plane 3: Plane enabled Plane 4: Links ok Plane 5: Links ok Plane 6: Links ok Plane 7: Links ok PFE #2 Plane 0: Plane enabled Plane 1: Plane enabled Plane 2: Plane enabled Plane 3: Plane enabled Plane 4: Links ok Plane 5: Links ok Plane 6: Links ok Plane 7: Links ok FPC 5 PFE #0 Plane 0: Plane enabled Plane 1: Plane enabled Plane 2: Plane enabled Plane 3: Plane enabled Plane 4: Plane enabled Plane 5: Unused Plane 6: Plane enabled Plane 7: Unused
In the show chassis fabric fpcs
command output, FPC 5 is the AS MCC.
The show chassis fabric plane
command displays the
output on an MX480 router with an AS MCC.
user@host>show chassis fabric plane Fabric management PLANE state Plane 0 Plane state: ACTIVE FPC 2 PFE 0 :Links ok FPC 4 PFE 0 :Links ok PFE 2 :Links ok FPC 5 PFE 0 :Links ok Plane 1 Plane state: ACTIVE FPC 2 PFE 0 :Links ok FPC 4 PFE 0 :Links ok PFE 2 :Links ok FPC 5 PFE 0 :Links ok Plane 2 Plane state: ACTIVE FPC 2 PFE 0 :Links ok FPC 4 PFE 0 :Links ok PFE 2 :Links ok FPC 5 PFE 0 :Links ok Plane 3 Plane state: ACTIVE FPC 2 PFE 0 :Links ok FPC 4 PFE 0 :Links ok PFE 2 :Links ok FPC 5 PFE 0 :Links ok Plane 4 Plane state: ACTIVE FPC 2 PFE 0 :Links ok FPC 4 PFE 0 :Links ok PFE 2 :Links ok FPC 5 PFE 0 :Links ok Plane 5 Plane state: ACTIVE FPC 2 PFE 0 :Links ok FPC 4 PFE 0 :Links ok PFE 2 :Links ok FPC 5 PFE 0 :Unused Plane 6 Plane state: ACTIVE FPC 2 PFE 0 :Links ok FPC 4 PFE 0 :Links ok PFE 2 :Links ok FPC 5 PFE 0 :Links ok Plane 7 Plane state: ACTIVE FPC 2 PFE 0 :Links ok FPC 4 PFE 0 :Links ok PFE 2 :Links ok FPC 5 PFE 0 :Unused
In the show chassis fabric plane
output, FPC 5 is the AS MCC.
The term Unused
in the output for the show chassis fabric fpcs and show chassis
fabric plane command indicates that one fabric plane
from each pair that share physical links (1 and 5, and 3 and 7) is
inactive.
See Junos OS System Basics and Services Command Reference for more information.
Fabric Plane Management on MX304 Routers
The SFB on MX304 router supports the following functionalities:
• Fabric Hardening: Controls bandwidth degradation and prevents null route.
• Fabric Fault Management: Supported per plane. Fabric fault management per plane results in increased granularity, to identify, isolate, and repair faults.
Fabric Hardening Support and Plane Management on MX304 Routers
Fabric plane management incudes fabric hardening, that is the process to control bandwidth degradation and prevent a null route for data transmission.
MX304 routers have only one built-in SFB and line card MIC, MX304-LMIC16-BASE. The SFB has two PFEs. Each PFE supports 18 fabric planes (or sub-channels).
LMICs | Switch Fabric Boards Supported | Packet Forwarding Engines (PFEs) | Fabric Planes | Fabric Redundancy |
---|---|---|---|---|
MX304-LMIC16-BASE | 1 SFB | 2 PFE | 36 | No |
Failure or Fault | Default Action | Configurable Action |
---|---|---|
All planes of a PFE come down (due to training failures, destination timeouts or combination of both). | Affected PFE is disabled. | Log only, FPC offline, FPC restart, FPC restart and then offline. |
Multiple PFEs lose all 18 planes (number of PFEs are less than 50% in the chassis) |
Affected PFEs are disabled. | Log only, FPC offline, FPC restart, FPC restart and then offline. |
Combination PFEs are at fault. | Affected PFEs are disabled. | Log only, FPC offline, FPC restart, FPC restart and then offline. |
All 18 planes are offlined or more than 50% of the PFEs in the chassis have faults. | SFB restart and FPC restart. If the attempt fails, PFEs are disabled. | Ignore SFB restart, Ignore FPC restart. |
SFB Fatal error | SFB reset– attempts 3 times before giving up. | None |
The following key CLI commands are available for fabric hardening:
-
set chassis fpc slot-number fabric bandwidth-degradation percentage
—Configures the FPC to take a specific action once bandwidth degradation reaches a certain percentage to avoid causing a null route in the chassis. -
set chassis fabric degraded detection-enable
—Enables detection of an FPC with degraded fabric. -
set chassis fabric degraded action-fpc-restart-disable
—Disables line card restarts to limit recovery actions from a degraded fabric condition. -
Use the commands
show chassis fabric reachability detail
to see if any fabric hardening actions are taken. -
Use command
show chassis fabric degradation
to check bandwidth information. -
Use
show chassis fabric summary extended
andshow log chassisd
for log information.
Limitations
• MX304 routers have only one built-in SFB and one FPC. Hence there is no fabric redundancy support.
• SFB offline and online is not supported. The command request chassis sfb slot 0
{offline| online}
is not supported. You can control the operation of the
specified fabric planes by using the command request chassis fabric plane
plane_number {offline| online}
.
Signaling Neighboring Routers of Fabric Down on T640 and T1600 Routers
In JUNOS OS Release 10.4 and later, T640 and T1600 routers signal neighboring routers if they are unable to carry traffic due to all fabric planes being taken offline for one of the following reasons:
CLI or button press initiated offline state.
Automatically taken offline by the SPMB due to high temperature.
PIO errors or voltage errors detected by the SPMB CPU to the SIBs.
The following scenarios are not supported:
All PFEs get destination errors on all planes to all destinations, even with the Switch Interface Boards (SIBs) staying online.
Complete fabric loss caused by destination timeouts, with the SIBs still online.
When chassisd detects all fabric planes are down, the router reboots all the FPCs in the system. When the FPCs come back up, the interfaces will not be created again, since all the fabric planes are down.
Once the user diagnoses and fixes the cause of all fabric planes going down, the user must then online the SIBs. The SIB online process brings up the interfaces.
Fabric down signaling to neighboring routers offers the following benefits:
FPCs reboot when the control plane connection to the RE times out.
Extends a simple approach to reboot FPCs when the dataplane blacks out.
When the router transitions from a state where SIBs are online or spare to a state where there are no SIBs in online state, then all the FPCs in the system are rebooted.
An ERRMSG indicates if all fabric planes are down and the FPCs will be rebooted if any fabric planes do not come up in 2 minutes.
An ERRMSG indicates the reason for FPC reboot on fabric connectivity loss.
The chassisd daemon traces when an FPC comes online, but PIC attach is not done due to no fabric plane present.
A warning is issued in the CLI when the last fabric plane is taken offline, that FPCs will reboot. You will need to online the SIBs after fixing the cause of the SIBs not being online. When the first SIB goes online, and link training with the FPCs completes, the interfaces will be created.
Fabric down signaling to neighboring routers functionality is available by default, and no user configuration required to enable it.
No CLI commands or alarms are required for this feature. Alarms indicate an SIBs offline system state to the user.