Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

Example: Configuring EX4200 Switches for the QFX3000-M QFabric System Control Plane

 

This example shows you how to connect QFabric system components and configure the EX4200 switches used for the QFX3000-M QFabric system control plane network. Proper wiring of Director devices, Interconnect devices, and Node devices to the EX4200 switches, combined with a standard configuration, enables you to bring up the internal QFabric system management network and prepare your QFabric system for full operation.

Note

The EX4200 switch configuration is the same for both the copper-based and fiber-based QFX3000-M QFabric system control plane networks. Hence, a separate example for configuring EX4200 switches for the fiber-based control plane network is not provided.

However, because you cannot mix and match fiber and copper in the same control plane network, you must select only one type of control plane for each QFX3000-M QFabric system you install. The primary focus of this example is a copper-based control plane network. Before you use this example to configure a fiber-based control plane network, ensure that you have installed and wired the QFabric system hardware and EX4200 switches as required for a fiber-based control plane network (see QFX3000-M QFabric System Installation Overview).

Requirements

This example uses the following hardware and software components:

  • One QFX3000-M QFabric system containing:

    • Two QFX3100 Director devices with 1000BASE-T network modules installed

    • Two QFX3600-I Interconnect devices

    • Eight QFX3500 Node devices with a 1000BASE-T management board installed

  • Two EX4200-24T switches with SFP+ uplink module installed

  • Junos OS Release 13.2X52-D10 for the QFabric system components

  • Junos OS Release 12.3R6.6 for the EX Series switches

Before you begin:

Overview

The QFX3000-M QFabric system control plane network connects the Director group, Interconnect devices, and Node devices in a QFabric system across a pair of redundant EX4200 switches. By separating the management control plane from the data plane, the QFabric system can scale efficiently. The copper-based control plane network uses Gigabit Ethernet cabling and connections between components, and two 1-Gigabit Ethernet connections configured in a link aggregation group (LAG) between the redundant EX4200 switches.

Specific ports have been reserved on the EX4200 switches to connect to each of the QFabric system device types. Such design simplifies installation and facilitates timely deployment of a QFabric system. It also permits the use of a standard EX4200 switch configuration included as part of this example. The standard configuration can scale from the 8 Node devices shown in this example to a maximum of 16 Node devices.

Topology

Figure 1 shows the general port ranges where QFabric system devices must be connected to the EX4200 switches. For each EX4200 switch, connect ports 0 through 15 to Node devices, ports 16 through 19 to Interconnect devices, ports 20 through 23 to Director devices, and uplink ports 0 and 1 to the other control plane switch as an inter-switch LAG. Table 1 shows the details of the QFabric system component-to-EX4200 switch port mappings.

Figure 1: QFX3000-M QFabric System Control Plane—EX4200 Switch Port Ranges
QFX3000-M QFabric System Control Plane—EX4200
Switch Port Ranges
Caution
  • The control plane network within a QFabric system is a critical component of the system that should not be shared with other network traffic. In order to scale efficiently, the control plane network must be reserved for the QFabric system and its components. As a result, the ports of the QFabric system control plane must never be used for any purpose other than to transport QFabric system control plane traffic, and we neither recommend nor support the connection of other devices to the QFabric system control plane network.

  • Do not install Junos Space and AI-Scripts (AIS) on the control plane network EX4200 switches in a QFX3000-M QFabric system.

Table 1 shows the specific mappings of QFabric system control plane network ports from the QFabric system components to the EX4200 switches.

Note

The uplink ports 2 and 3 on the EX4200 switches are reserved for future use.

Table 1: QFX3000-M QFabric System Copper-Based Control Plane—QFabric Component-to-EX4200 Switch Port Mappings

EX4200 Switch 1 (EX0)

EX4200 Switch 2 (EX1)

QFabric System Component

Node0, management port C0 to port 0

(ge-0/0/0)

Node0, management port C1 to port 0

(ge-0/0/0)

Node device 0

Node1, management port C0 to port 1

(ge-0/0/1)

Node1, management port C1 to port 1

(ge-0/0/1)

Node device 1

Node2, management port C0 to port 2

(ge-0/0/2)

Node2, management port C1 to port 2

(ge-0/0/2)

Node device 2

Node3, management port C0 to port 3

(ge-0/0/3)

Node3, management port C1 to port 3

(ge-0/0/3)

Node device 3

Node4, management port C0 to port 4

(ge-0/0/4)

Node4, management port C1 to port 4

(ge-0/0/4)

Node device 4

Node5, management port C0 to port 5

(ge-0/0/5)

Node5, management port C1 to port 5

(ge-0/0/5)

Node device 5

Node6, management port C0 to port 6

(ge-0/0/6)

Node6, management port C1 to port 6

(ge-0/0/6)

Node device 6

Node7, management port C0 to port 7

(ge-0/0/7)

Node7, management port C1 to port 7

(ge-0/0/7)

Node device 7

Node8, management port C0 to port 8

(ge-0/0/8)

Node8, management port C1 to port 8

(ge-0/0/8)

Node device 8

Node9, management port C0 to port 9

(ge-0/0/9)

Node9, management port C1 to port 9

(ge-0/0/9)

Node device 9

Node10, management port C0 to port 10

(ge-0/0/10)

Node10, management port C1 to port 10

(ge-0/0/10)

Node device 10

Node11, management port C0 to port 11

(ge-0/0/11)

Node11, management port C1 to port 11

(ge-0/0/11)

Node device 11

Node12, management port C0 to port 12

(ge-0/0/12)

Node12, management port C1 to port 12

(ge-0/0/12)

Node device 12

Node13, management port C0 to port 13

(ge-0/0/13)

Node13, management port C1 to port 13

(ge-0/0/13)

Node device 13

Node14, management port C0 to port 14

(ge-0/0/14)

Node14, management port C1 to port 14

(ge-0/0/14)

Node device 14

Node15, management port C0 to port 15

(ge-0/0/15)

Node15, management port C1 to port 15

(ge-0/0/15)

Node device 15

IC0, management port C0 to port 16

(ge-0/0/16)

IC0, management port C1 to port 16

(ge-0/0/16)

Interconnect device 0

IC1, management port C0 to port 17

(ge-0/0/17)

IC1, management port C1 to port 17

(ge-0/0/17)

Interconnect device 1

IC2, management port C0 to port 18

(ge-0/0/18)

IC2, management port C1 to port 18

(ge-0/0/18)

Interconnect device 2

IC3, management port C0 to port 19

(ge-0/0/19)

IC3, management port C1 to port 19

(ge-0/0/19)

Interconnect device 3

DG0 module 0, port 0 to port 20

(ge-0/0/20)

DG0 module 1, port 0 to port 20

(ge-0/0/20)

Director device 0

DG0 module 0, port 1 to port 21

(ge-0/0/21)

DG0 module 1, port 1 to port 21

(ge-0/0/21)

Director device 0

DG1 module 0, port 0 to port 22

(ge-0/0/22)

DG1 module 1, port 0 to port 22

(ge-0/0/22)

Director device 1

DG1 module 0, port 1 to port 23

(ge-0/0/23)

DG1 module 1, port 1 to port 23

(ge-0/0/23)

Director device 1

EX0, uplink port 0 to EX1, uplink port 0

(ge-0/1/0)

EX1, uplink port 0 to EX0, uplink port 0

(ge-0/1/0)

Inter-EX4200 switch LAG

EX0, uplink port 1 to EX1, uplink port 1

(ge-0/1/1)

EX1, uplink port 1 to EX0, uplink port 1

(ge-0/1/1)

Inter-EX4200 switch LAG

Reserved

Uplink port 2 (ge-0/1/2)

Reserved

Uplink port 2 (ge-0/1/2)

Future use

Reserved

Uplink port 3 (ge-0/1/3)

Reserved

Uplink port 3 (ge-0/1/3)

Future use

Next, connect the Director devices to the EX4200 switches. In general, you want to accomplish the following:

  • Connect two ports from one network module in a Director device to the first EX4200 switch, and two ports from the second network module to the second EX4200 switch.

  • Connect the Director devices to each other and create a Director group. You can use either straight-through RJ-45 patch cables or crossover cables, because the Director devices contain autosensing modules. Connect one port from each network module on the first Director device to one port in each network module on the second Director device.

Figure 2 shows the specific ports on the Director group that you must connect to the EX4200 switches and interconnect between the Director devices.

Figure 2: QFX3000-M QFabric System Control Plane—Director Group to EX4200 Switch Connections
QFX3000-M QFabric System Control Plane—Director
Group to EX4200 Switch Connections

In this specific example, connect ports 0 and 1 from module 0 on Director device DG0 to ports 20 and 21 on EX4200 switch EX0 (ge-0/0/20 and ge-0/0/21), and connect ports 0 and 1 from module 1 to ports 20 and 21 on the second EX4200 switch EX1 (ge-0/0/20 and ge-0/0/21).

For Director device DG1, connect ports 0 and 1 from module 0 to ports 22 and 23 on EX4200 switch EX0 (ge-0/0/22 and ge-0/0/23), and connect ports 0 and 1 from module 1 to ports 22 and 23 on the second EX4200 switch EX1 (ge-0/0/22 and ge-0/0/23).

To form the Director group, connect port 3 on module 0 on Director device DG0 to port 3 on module 0 on Director device DG1. Similarly, connect port 3 on module 1 on Director device DG0 to port 3 on module 1 on Director device DG1. Table 2 shows the port mappings for the Director group in this example.

Table 2: Director Group Port Mappings

Director Device

EX4200 Switch EX0

EX4200 Switch EX1

DG0

  • DG0 module 0, port 0 to port 20 on EX0

    (ge-0/0/20)

  • DG0 module 0, port 1 to port 21 on EX0

    (ge-0/0/21)

  • DG0 module 0, port 3 to module 0, port 3 on DG1

  • DG0 module 1, port 0 to port 20 on EX1

    (ge-0/0/20)

  • DG0 module 1, port 1 to port 21 on EX1

    (ge-0/0/21)

  • DG0 module 1, port 3 to module 1, port 3 on DG1

DG1

  • DG1 module 0, port 0 to port 22 on EX0

    (ge-0/0/22)

  • DG1 module 0, port 1 to port 23 on EX0

    (ge-0/0/23)

  • DG1 module 0, port 3 to module 0, port 3 on DG0

  • DG1 module 1, port 0 to port 22 on EX1

    (ge-0/0/22)

  • DG1 module 1, port 1 to port 23 on EX1

    (ge-0/0/23)

  • DG1 module 1, port 3 to module 1, port 3 on DG0

In the software, the ports of each network module on a Director device are reversed, numbered from right to left, and incremented sequentially across modules. If you issue interface operational commands directly on the Director device, note the following port mappings as shown in Table 3:

Table 3: Hardware to Software Port Mappings for Director Device Network Modules

Network Module

Port 0

Port 1

Port 2

Port 3

Module 0

eth5

eth4

eth3

eth2

Module 1

eth9

eth8

eth7

eth6

Figure 3 shows the specific ports on the QFX3600-I Interconnect devices that you must connect to the EX4200 switches. In general, connect the first management port in an Interconnect device to the first EX4200 switch, and the second management port to the second EX4200 switch.

Figure 3: QFX3000-M QFabric System Control Plane—Interconnect Device to EX4200 Switch Connections
QFX3000-M QFabric System Control Plane—Interconnect
Device to EX4200 Switch Connections

In this specific example, for both Interconnect devices IC0 and IC1, connect management port C0 to EX4200 switches EX0 and EX1 and management port C1 to EX4200 switches EX0 and EX1. Connect the management port C0 cables to port 16 on EX4200 switches EX0 and EX1 (ge-0/0/16), and connect the management port C1 cables to port 17 on EX4200 switches EX0 and EX1 (ge-0/0/17). Table 4 shows the port mappings for the Node devices in this example.

Table 4: Interconnect Device Port Mappings

Interconnect Device

EX4200 Switch EX0

EX4200 Switch EX1

IC0

IC0, management port C0 to port 16

(ge-0/0/16)

IC0, management port C1 to port 16

(ge-0/0/16)

IC1

IC1, management port C0 to port 17

(ge-0/0/17)

IC1, management port C1 to port 17

(ge-0/0/17)

Figure 4, Figure 5, and Figure 6 show the specific ports on the Node devices that you must connect to the EX4200 switches when using a copper-based control plane. In general, connect the first management port from a Node device to the first EX4200 switch, and the second management port to the second EX4200 switch.

Figure 4: QFX3000-M QFabric System Control Plane—QFX3500 Node Device to EX4200 Switch Connections
QFX3000-M QFabric System Control Plane—QFX3500
Node Device to EX4200 Switch Connections
Figure 5: QFX3000-M QFabric System Control Plane—QFX3600 Node Device to EX4200 Switch Connections
QFX3000-M QFabric System Control Plane—QFX3600
Node Device to EX4200 Switch Connections
Figure 6: QFX3000-M QFabric System Control Plane—QFX5100 Node Device to EX4200 Switch Connections
QFX3000-M QFabric System Control Plane—QFX5100
Node Device to EX4200 Switch Connections

When implementing a fiber-based control plane, refer to Figure 7, Figure 8, and Figure 9 for the proper control plane connections.

Figure 7: QFX3000-M QFabric System Fiber-Based Control Plane—QFX3500 Node Device to EX4200 Switch Connections
QFX3000-M QFabric System Fiber-Based Control
Plane—QFX3500 Node Device to EX4200 Switch Connections
Figure 8: QFX3000-M QFabric System Fiber-Based Control Plane—QFX3600 Node Device to EX4200 Switch Connections
QFX3000-M QFabric System Fiber-Based Control
Plane—QFX3600 Node Device to EX4200 Switch Connections
Figure 9: QFX3000-M QFabric System Fiber-Based Control Plane—QFX5100 Node Device to EX4200 Switch Connections
QFX3000-M QFabric System Fiber-Based Control
Plane—QFX5100 Node Device to EX4200 Switch Connections

In this specific example, for Node device Node0, connect management port C0 (also known as me5) to EX4200 switch EX0 port 0 (ge-0/0/0), and connect management port C1 (also known as me6) to the second EX4200 switch EX1 port 0 (ge-0/0/0).

For the remaining seven Node devices, connect management port C0 to the ge-0/0/X port on EX4200 switch EX0 that matches the Node device number. Similarly, connect management port C1 to the port on the second EX4200 switch EX1 that matches the Node device number. For example, you would connect Node device Node5 to port 5 (ge-0/0/5). Table 5 shows the full set of port mappings for the Node devices in this example.

Table 5: Node Device to EX4200 Switch Port Mappings

Node Device

EX4200 Switch EX0

EX4200 Switch EX1

Node0

Node0, management port C0 to port 0

(ge-0/0/0)

Node0, management port C1 to port 0

(ge-0/0/0)

Node1

Node1, management port C0 to port 1

(ge-0/0/1)

Node1, management port C1 to port 1

(ge-0/0/1)

Node2

Node2, management port C0 to port 2

(ge-0/0/2)

Node2, management port C1 to port 2

(ge-0/0/2)

Node3

Node3, management port C0 to port 3

(ge-0/0/3)

Node3, management port C1 to port 3

(ge-0/0/3)

Node4

Node4, management port C0 to port 4

(ge-0/0/4)

Node4, management port C1 to port 4

(ge-0/0/4)

Node5

Node5, management port C0 to port 5

(ge-0/0/5)

Node5, management port C1 to port 5

(ge-0/0/5)

Node6

Node6, management port C0 to port 6

(ge-0/0/6)

Node6, management port C1 to port 6

(ge-0/0/6)

Node7

Node7, management port C0 to port 7

(ge-0/0/7)

Node7, management port C1 to port 7

(ge-0/0/7)

Figure 10 shows the specific uplink ports on the first EX4200 switch that you must connect to the second EX4200 switch. These connections create a link aggregation group (LAG) that provides redundancy and resiliency for the EX4200 switch portion of the control plane. In general, connect each 1-Gigabit Ethernet uplink port from the first EX4200 switch to the corresponding 1-Gigabit Ethernet uplink port on the second EX4200 switch.

Figure 10: QFX3000-M QFabric System Control Plane—Inter-EX4200 Switch LAG Connections
QFX3000-M QFabric System Control Plane—Inter-EX4200
Switch LAG Connections

In this specific example, for EX4200 switch EX0, connect uplink port 0 (ge-0/1/0) to EX4200 switch EX1 uplink port 0 (ge-0/1/0). Then connect uplink port 1 (ge-0/1/1) on EX4200 switch EX0 to uplink port 1 (ge-0/1/1) on EX4200 switch EX1.

Table 6 shows the port mappings for the EX4200 switch LAG connections in this example.

Table 6: EX4200 Switch LAG Port Mappings

EX0 and EX1

EX0

EX1

Uplink port 0

ge-0/1/0 to ge-0/1/0

ge-0/1/0 to ge-0/1/0

Uplink port 1

ge-0/1/1 to ge-0/1/1

ge-0/1/1 to ge-0/1/1

Configuration

CLI Quick Configuration

To configure the QFX3000-M QFabric system control plane EX4200 switches quickly, copy the following commands, paste them in a text file, remove any line breaks, change any details necessary to match your network, and then copy and paste the commands into the EX4200 switch CLI at the [edit] hierarchy level.

Step-by-Step Procedure

The following example requires that you navigate various levels in the configuration hierarchy. For instructions on how to do that, see Using the CLI Editor in Configuration Mode in the CLI User Guide.

To configure a EX4200 switch for the QFX3000-M QFabric system control plane network:

  1. Create a configuration group called qfabric to define global QFabric system control plane properties. Set up the number of aggregated Ethernet devices, configure alarm and LCD management, activate loop prevention and storm control, specify a global VLAN (VLAN 100) and 802.1q tunneling, define options for aggregated Ethernet interfaces, and apply the qfabric group settings to the configuration.
  2. Configure interfaces for the QFabric system control plane network. Enable the EX4200 switch SFP+ uplink module for 1-Gigabit Ethernet operation. Set the interface ranges where Node devices (0 through 15), Interconnect devices (16 and 17), and Director devices (20 through 23) connect to the control plane network through the EX4200 switches. Configure the inter-EX4200 switch LAG connections for the ae2 interface and apply the qfabric-int and qfabric-ae configuration groups to the aggregated Ethernet interfaces (ae0 and ae1) for the Director devices.
  3. Enable class of service (CoS) for the QFabric system control plane network. Establish forwarding classes, priorities, scheduler maps, classifiers, and queues for three types of traffic: control traffic, interdevice traffic, and best-effort traffic.
  4. Configure settings to enable the EX4200 switches to interoperate with your management network. Set a hostname, system services (such as Telnet), system log thresholds, management interface parameters, default routes, and any additional preferences you might have.

Results

To view the configuration, issue the show command in configuration mode or the show configuration command in operational mode. If the output does not display the intended configuration, repeat the configuration instructions in this example to correct it.

The following configuration is the standard configuration that applies universally to both EX4200 switches in your QFabric system control plane network.

The following portion of the configuration applies to the specific requirements of your management network. Modify this section to meet the needs of your network.

To verify the syntax of your configuration prior to committing it, enter commit check from configuration mode. If you are done configuring the device, enter commit from configuration mode.

Verification

Confirm that the EX4200 switch configuration is working properly.

Verifying the QFX3000-M QFabric System Control Plane—EX4200 Switch EX0

Purpose

Verify that the control plane is properly connected on your first EX4200 switch.

Action

Connect to the Junos OS CLI of EX4200 switch EX0, either from your management network or from the console port of the switch. In operational mode, enter the show interfaces terse command.

Sample Output

user@ex0> show interfaces terse

Meaning

In the output of the show interfaces terse command, if all interfaces that connect to the QFabric system devices are listed as up (such as ge-0/0/16 and ge-0/0/17 for the Interconnect devices; ge-0/0/20 through ge-0/0/23 for the Director devices; ge-0/0/0 through ge-0/0/7 for the Node devices; and ge-0/1/0 and ge-0/1/1 for the inter-EX4200 switch connections), the control plane is properly connected.

Verifying the QFX3000-M QFabric System Control Plane—EX4200 Switch EX1

Purpose

Verify that the control plane is properly connected on your second EX4200 switch.

Action

Connect to the Junos OS CLI of EX4200 switch EX1, either from your management network or from the console port of the switch. In operational mode, enter the show interfaces terse command.

Sample Output

user@ex1> show interfaces terse

Meaning

In the output of the show interfaces terse command, if all interfaces that connect to the QFabric system devices are listed as up (such as ge-0/0/16 and ge-0/0/17 for the Interconnect devices; ge-0/0/20 through ge-0/0/23 for the Director devices; ge-0/0/0 through ge-0/0/7 for the Node devices; and ge-0/1/0 and ge-0/1/1 for the inter-EX4200 switch connections), the control plane is properly connected.