Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

 
 

Example: Configuring EX Series Switches for the QFX3000-M QFabric System Control Plane

This example shows you how to connect QFabric system components and configure EX4200 switches used for the QFX3000-M QFabric system control plane network. Proper wiring of Director devices, Interconnect devices, and Node devices to the EX Series switches, combined with a standard configuration, enables you to bring up the internal QFabric system management network and prepare your QFabric system for full operation.

Note:

EX4200 or EX4300 switch configuration is the same for both the copper-based and fiber-based QFX3000-M QFabric system control plane networks. Hence, a separate example for configuring EX4300 switches or the fiber-based control plane network is not provided.

However, because you cannot mix and match fiber and copper in the same control plane network, you must select only one type of control plane for each QFX3000-M QFabric system you install. The primary focus of this example is a copper-based control plane network with EX4200-24T switches. Before you use this example to configure a fiber-based control plane network with EX4200 switches, ensure that you have installed and wired the QFabric system hardware and EX4200 switches as required for a fiber-based control plane network (see QFX3000-M QFabric System Installation Overview). Before you use this example to configure a copper or fiber-based control plane network with EX4300 switches, ensure that you have installed and wired the QFabric system hardware and EX4300 switches as required in Installing and Connecting an EX4300 Switch.

Requirements

This example uses the following hardware and software components:

  • One QFX3000-M QFabric system containing:

    • Two QFX3100 Director devices with 1000BASE-T network modules installed

    • Two QFX3600-I Interconnect devices

    • Eight QFX3500 Node devices with a 1000BASE-T management board installed

  • Two EX4200-24T switches with SFP+ uplink module installed

    Note:

    Instead of two EX4200-24T switches, you can use two EX4300-48T switches for a copper-based control plane network. For a fiber-based control plane network, you can use two EX4200-24F switches or two EX4300-48P switches.

  • Junos OS Release 13.2X52-D10 for the QFabric system components

  • Junos OS Release 12.3R6.6 for the EX Series switches

Before you begin:

Overview

The QFX3000-M QFabric system control plane network connects the Director group, Interconnect devices, and Node devices in a QFabric system across a pair of redundant EX Series switches. By separating the management control plane from the data plane, the QFabric system can scale efficiently. The copper-based control plane network uses Gigabit Ethernet cabling and connections between components, and two 1-Gigabit Ethernet connections configured in a link aggregation group (LAG) between the redundant EX Series switches.

Specific ports have been reserved on the EX Series switches to connect to each of the QFabric system device types. Such design simplifies installation and facilitates timely deployment of a QFabric system. It also permits the use of a standard EX Series switch configuration included as part of this example. The standard configuration can scale from the 8 Node devices shown in this example to a maximum of 16 Node devices.

Topology

Figure 1 shows the general port ranges where QFabric system devices must be connected to the EX Series switches. For each EX Series switch, connect ports 0 through 15 to Node devices, ports 16 through 19 to Interconnect devices, ports 20 through 23 to Director devices, and uplink ports 0 and 1 to the other control plane switch as an inter-switch LAG. Table 1 shows the details of the QFabric system component-to-EX Series switch port mappings.

Figure 1: QFX3000-M QFabric System Control Plane—EX Series Switch Port RangesQFX3000-M QFabric System Control Plane—EX Series Switch Port Ranges
CAUTION:
  • The control plane network within a QFabric system is a critical component of the system that should not be shared with other network traffic. In order to scale efficiently, the control plane network must be reserved for the QFabric system and its components. As a result, the ports of the QFabric system control plane must never be used for any purpose other than to transport QFabric system control plane traffic, and we neither recommend nor support the connection of other devices to the QFabric system control plane network.

  • Do not install Junos Space and AI-Scripts (AIS) on the control plane network EX Series switches in a QFX3000-M QFabric system.

Table 1 shows the specific mappings of QFabric system control plane network ports from the QFabric system components to the EX Series switches.

Note:

The uplink ports 2 and 3 on the EX Series switches are reserved for future use.

Table 1: QFX3000-M QFabric System Copper-Based Control Plane—QFabric Component-to-EX Series Switch Port Mappings

EX Series Switch 1 (EX0)

EX Series Switch 2 (EX1)

QFabric System Component

Node0, management port C0 to port 0

(ge-0/0/0)

Node0, management port C1 to port 0

(ge-0/0/0)

Node device 0

Node1, management port C0 to port 1

(ge-0/0/1)

Node1, management port C1 to port 1

(ge-0/0/1)

Node device 1

Node2, management port C0 to port 2

(ge-0/0/2)

Node2, management port C1 to port 2

(ge-0/0/2)

Node device 2

Node3, management port C0 to port 3

(ge-0/0/3)

Node3, management port C1 to port 3

(ge-0/0/3)

Node device 3

Node4, management port C0 to port 4

(ge-0/0/4)

Node4, management port C1 to port 4

(ge-0/0/4)

Node device 4

Node5, management port C0 to port 5

(ge-0/0/5)

Node5, management port C1 to port 5

(ge-0/0/5)

Node device 5

Node6, management port C0 to port 6

(ge-0/0/6)

Node6, management port C1 to port 6

(ge-0/0/6)

Node device 6

Node7, management port C0 to port 7

(ge-0/0/7)

Node7, management port C1 to port 7

(ge-0/0/7)

Node device 7

Node8, management port C0 to port 8

(ge-0/0/8)

Node8, management port C1 to port 8

(ge-0/0/8)

Node device 8

Node9, management port C0 to port 9

(ge-0/0/9)

Node9, management port C1 to port 9

(ge-0/0/9)

Node device 9

Node10, management port C0 to port 10

(ge-0/0/10)

Node10, management port C1 to port 10

(ge-0/0/10)

Node device 10

Node11, management port C0 to port 11

(ge-0/0/11)

Node11, management port C1 to port 11

(ge-0/0/11)

Node device 11

Node12, management port C0 to port 12

(ge-0/0/12)

Node12, management port C1 to port 12

(ge-0/0/12)

Node device 12

Node13, management port C0 to port 13

(ge-0/0/13)

Node13, management port C1 to port 13

(ge-0/0/13)

Node device 13

Node14, management port C0 to port 14

(ge-0/0/14)

Node14, management port C1 to port 14

(ge-0/0/14)

Node device 14

Node15, management port C0 to port 15

(ge-0/0/15)

Node15, management port C1 to port 15

(ge-0/0/15)

Node device 15

IC0, management port C0 to port 16

(ge-0/0/16)

IC0, management port C1 to port 16

(ge-0/0/16)

Interconnect device 0

IC1, management port C0 to port 17

(ge-0/0/17)

IC1, management port C1 to port 17

(ge-0/0/17)

Interconnect device 1

IC2, management port C0 to port 18

(ge-0/0/18)

IC2, management port C1 to port 18

(ge-0/0/18)

Interconnect device 2

IC3, management port C0 to port 19

(ge-0/0/19)

IC3, management port C1 to port 19

(ge-0/0/19)

Interconnect device 3

DG0 module 0, port 0 to port 20

(ge-0/0/20)

DG0 module 1, port 0 to port 20

(ge-0/0/20)

Director device 0

DG0 module 0, port 1 to port 21

(ge-0/0/21)

DG0 module 1, port 1 to port 21

(ge-0/0/21)

Director device 0

DG1 module 0, port 0 to port 22

(ge-0/0/22)

DG1 module 1, port 0 to port 22

(ge-0/0/22)

Director device 1

DG1 module 0, port 1 to port 23

(ge-0/0/23)

DG1 module 1, port 1 to port 23

(ge-0/0/23)

Director device 1

EX0, uplink port 0 to EX1, uplink port 0

(ge-0/1/0)

EX1, uplink port 0 to EX0, uplink port 0

(ge-0/1/0)

Inter-EX Series switch LAG

EX0, uplink port 1 to EX1, uplink port 1

(ge-0/1/1)

EX1, uplink port 1 to EX0, uplink port 1

(ge-0/1/1)

Inter-EX Series switch LAG

Reserved

Uplink port 2 (ge-0/1/2)

Reserved

Uplink port 2 (ge-0/1/2)

Future use

Reserved

Uplink port 3 (ge-0/1/3)

Reserved

Uplink port 3 (ge-0/1/3)

Future use

Next, connect the Director devices to the EX Series switches. In general, you want to accomplish the following:

  • Connect two ports from one network module in a Director device to the first EX Series switch, and two ports from the second network module to the second EX Series switch.

  • Connect the Director devices to each other and create a Director group. You can use either straight-through RJ-45 patch cables or crossover cables, because the Director devices contain autosensing modules. Connect one port from each network module on the first Director device to one port in each network module on the second Director device.

Figure 2 shows the specific ports on the Director group that you must connect to the EX Series switches and interconnect between the Director devices.

Figure 2: QFX3000-M QFabric System Control Plane—Director Group to EX Series Switch ConnectionsQFX3000-M QFabric System Control Plane—Director Group to EX Series Switch Connections

In this specific example, connect ports 0 and 1 from module 0 on Director device DG0 to ports 20 and 21 on EX Series switch EX0 (ge-0/0/20 and ge-0/0/21), and connect ports 0 and 1 from module 1 to ports 20 and 21 on the second EX Series switch EX1 (ge-0/0/20 and ge-0/0/21).

For Director device DG1, connect ports 0 and 1 from module 0 to ports 22 and 23 on EX Series switch EX0 (ge-0/0/22 and ge-0/0/23), and connect ports 0 and 1 from module 1 to ports 22 and 23 on the second EX Series switch EX1 (ge-0/0/22 and ge-0/0/23).

To form the Director group, connect port 3 on module 0 on Director device DG0 to port 3 on module 0 on Director device DG1. Similarly, connect port 3 on module 1 on Director device DG0 to port 3 on module 1 on Director device DG1. Table 2 shows the port mappings for the Director group in this example.

Table 2: Director Group Port Mappings

Director Device

EX Series Switch EX0

EX Series Switch EX1

DG0

  • DG0 module 0, port 0 to port 20 on EX0

    (ge-0/0/20)

  • DG0 module 0, port 1 to port 21 on EX0

    (ge-0/0/21)

  • DG0 module 0, port 3 to module 0, port 3 on DG1

  • DG0 module 1, port 0 to port 20 on EX1

    (ge-0/0/20)

  • DG0 module 1, port 1 to port 21 on EX1

    (ge-0/0/21)

  • DG0 module 1, port 3 to module 1, port 3 on DG1

DG1

  • DG1 module 0, port 0 to port 22 on EX0

    (ge-0/0/22)

  • DG1 module 0, port 1 to port 23 on EX0

    (ge-0/0/23)

  • DG1 module 0, port 3 to module 0, port 3 on DG0

  • DG1 module 1, port 0 to port 22 on EX1

    (ge-0/0/22)

  • DG1 module 1, port 1 to port 23 on EX1

    (ge-0/0/23)

  • DG1 module 1, port 3 to module 1, port 3 on DG0

In the software, the ports of each network module on a Director device are reversed, numbered from right to left, and incremented sequentially across modules. If you issue interface operational commands directly on the Director device, note the following port mappings as shown in Table 3:

Table 3: Hardware to Software Port Mappings for Director Device Network Modules

Network Module

Port 0

Port 1

Port 2

Port 3

Module 0

eth5

eth4

eth3

eth2

Module 1

eth9

eth8

eth7

eth6

Figure 3 shows the specific ports on the QFX3600-I Interconnect devices that you must connect to the EX Series switches. In general, connect the first management port in an Interconnect device to the first EX Series switch, and the second management port to the second EX Series switch.

Figure 3: QFX3000-M QFabric System Control Plane—Interconnect Device to EX Series Switch ConnectionsQFX3000-M QFabric System Control Plane—Interconnect Device to EX Series Switch Connections

In this specific example, for both Interconnect devices IC0 and IC1, connect management port C0 to EX Series switches EX0 and EX1 and management port C1 to EX Series switches EX0 and EX1. Connect the management port C0 cables to port 16 on EX Series switches EX0 and EX1 (ge-0/0/16), and connect the management port C1 cables to port 17 on EX Series switches EX0 and EX1 (ge-0/0/17). Table 4 shows the port mappings for the Node devices in this example.

Table 4: Interconnect Device Port Mappings

Interconnect Device

EX Series Switch EX0

EX Series Switch EX1

IC0

IC0, management port C0 to port 16

(ge-0/0/16)

IC0, management port C1 to port 16

(ge-0/0/16)

IC1

IC1, management port C0 to port 17

(ge-0/0/17)

IC1, management port C1 to port 17

(ge-0/0/17)

Figure 4, Figure 5, and Figure 6 show the specific ports on the Node devices that you must connect to the EX Series switches when using a copper-based control plane. In general, connect the first management port from a Node device to the first EX Series switch, and the second management port to the second EX Series switch.

Figure 4: QFX3000-M QFabric System Control Plane—QFX3500 Node Device to EX Series Switch ConnectionsQFX3000-M QFabric System Control Plane—QFX3500 Node Device to EX Series Switch Connections
Figure 5: QFX3000-M QFabric System Control Plane—QFX3600 Node Device to EX Series Switch ConnectionsQFX3000-M QFabric System Control Plane—QFX3600 Node Device to EX Series Switch Connections
Figure 6: QFX3000-M QFabric System Control Plane—QFX5100 Node Device to EX Series Switch ConnectionsQFX3000-M QFabric System Control Plane—QFX5100 Node Device to EX Series Switch Connections

When implementing a fiber-based control plane, refer to Figure 7, Figure 8, and Figure 9 for the proper control plane connections.

Figure 7: QFX3000-M QFabric System Fiber-Based Control Plane—QFX3500 Node Device to EX Series Switch ConnectionsQFX3000-M QFabric System Fiber-Based Control Plane—QFX3500 Node Device to EX Series Switch Connections
Figure 8: QFX3000-M QFabric System Fiber-Based Control Plane—QFX3600 Node Device to EX Series Switch ConnectionsQFX3000-M QFabric System Fiber-Based Control Plane—QFX3600 Node Device to EX Series Switch Connections
Figure 9: QFX3000-M QFabric System Fiber-Based Control Plane—QFX5100 Node Device to EX Series Switch ConnectionsQFX3000-M QFabric System Fiber-Based Control Plane—QFX5100 Node Device to EX Series Switch Connections

In this specific example, for Node device Node0, connect management port C0 (also known as me5) to EX Series switch EX0 port 0 (ge-0/0/0), and connect management port C1 (also known as me6) to the second EX Series switch EX1 port 0 (ge-0/0/0).

For the remaining seven Node devices, connect management port C0 to the ge-0/0/X port on EX Series switch EX0 that matches the Node device number. Similarly, connect management port C1 to the port on the second EX Series switch EX1 that matches the Node device number. For example, you would connect Node device Node5 to port 5 (ge-0/0/5). Table 5 shows the full set of port mappings for the Node devices in this example.

Table 5: Node Device to EX Series Switch Port Mappings

Node Device

EX Series Switch EX0

EX Series Switch EX1

Node0

Node0, management port C0 to port 0

(ge-0/0/0)

Node0, management port C1 to port 0

(ge-0/0/0)

Node1

Node1, management port C0 to port 1

(ge-0/0/1)

Node1, management port C1 to port 1

(ge-0/0/1)

Node2

Node2, management port C0 to port 2

(ge-0/0/2)

Node2, management port C1 to port 2

(ge-0/0/2)

Node3

Node3, management port C0 to port 3

(ge-0/0/3)

Node3, management port C1 to port 3

(ge-0/0/3)

Node4

Node4, management port C0 to port 4

(ge-0/0/4)

Node4, management port C1 to port 4

(ge-0/0/4)

Node5

Node5, management port C0 to port 5

(ge-0/0/5)

Node5, management port C1 to port 5

(ge-0/0/5)

Node6

Node6, management port C0 to port 6

(ge-0/0/6)

Node6, management port C1 to port 6

(ge-0/0/6)

Node7

Node7, management port C0 to port 7

(ge-0/0/7)

Node7, management port C1 to port 7

(ge-0/0/7)

Figure 10 shows the specific uplink ports on the first EX Series switch that you must connect to the second EX Series switch. These connections create a link aggregation group (LAG) that provides redundancy and resiliency for the EX Series switch portion of the control plane. In general, connect each 1-Gigabit Ethernet uplink port from the first EX Series switch to the corresponding 1-Gigabit Ethernet uplink port on the second EX Series switch.

Figure 10: QFX3000-M QFabric System Control Plane—Inter-EX Series Switch LAG ConnectionsQFX3000-M QFabric System Control Plane—Inter-EX Series Switch LAG Connections

In this specific example, for EX Series switch EX0, connect uplink port 0 (ge-0/1/0) to EX Series switch EX1 uplink port 0 (ge-0/1/0). Then connect uplink port 1 (ge-0/1/1) on EX Series switch EX0 to uplink port 1 (ge-0/1/1) on EX Series switch EX1.

Table 6 shows the port mappings for the EX Series switch LAG connections in this example.

Table 6: EX Series Switch LAG Port Mappings

EX0 and EX1

EX0

EX1

Uplink port 0

ge-0/1/0 to ge-0/1/0

ge-0/1/0 to ge-0/1/0

Uplink port 1

ge-0/1/1 to ge-0/1/1

ge-0/1/1 to ge-0/1/1

Configuration

Procedure

CLI Quick Configuration

To configure the QFX3000-M QFabric system control plane EX Series switches quickly, copy the following commands, paste them in a text file, remove any line breaks, change any details necessary to match your network, and then copy and paste the commands into the EX Series switch CLI at the [edit] hierarchy level.

Step-by-Step Procedure

The following example requires that you navigate various levels in the configuration hierarchy. For instructions on how to do that, see Using the CLI Editor in Configuration Mode in the CLI User Guide.

To configure an EX Series switch for the QFX3000-M QFabric system control plane network:

  1. Create a configuration group called qfabric to define global QFabric system control plane properties. Set up the number of aggregated Ethernet devices, configure alarm and LCD management, activate loop prevention and storm control, specify a global VLAN (VLAN 100) and 802.1q tunneling, define options for aggregated Ethernet interfaces, and apply the qfabric group settings to the configuration.

  2. Configure interfaces for the QFabric system control plane network. Enable the EX Series switch SFP+ uplink module for 1-Gigabit Ethernet operation. Set the interface ranges where Node devices (0 through 15), Interconnect devices (16 and 17), and Director devices (20 through 23) connect to the control plane network through the EX Series switches. Configure the inter-EX Series switch LAG connections for the ae2 interface and apply the qfabric-int and qfabric-ae configuration groups to the aggregated Ethernet interfaces (ae0 and ae1) for the Director devices.

  3. Enable class of service (CoS) for the QFabric system control plane network. Establish forwarding classes, priorities, scheduler maps, classifiers, and queues for three types of traffic: control traffic, interdevice traffic, and best-effort traffic.

  4. Configure settings to enable the EX Series switches to interoperate with your management network. Set a hostname, system services (such as Telnet), system log thresholds, management interface parameters, default routes, and any additional preferences you might have.

Results

To view the configuration, issue the show command in configuration mode or the show configuration command in operational mode. If the output does not display the intended configuration, repeat the configuration instructions in this example to correct it.

The following configuration is the standard configuration that applies universally to both EX Series switches in your QFabric system control plane network.

The following portion of the configuration applies to the specific requirements of your management network. Modify this section to meet the needs of your network.

To verify the syntax of your configuration prior to committing it, enter commit check from configuration mode. If you are done configuring the device, enter commit from configuration mode.

Verification

Confirm that the EX Series switch configuration is working properly.

Verifying the QFX3000-M QFabric System Control Plane—EX Series Switch EX0

Purpose

Verify that the control plane is properly connected on your first EX Series switch.

Action

Connect to the Junos OS CLI of EX Series switch EX0, either from your management network or from the console port of the switch. In operational mode, enter the show interfaces terse command.

Sample Output
command-name

Meaning

In the output of the show interfaces terse command, if all interfaces that connect to the QFabric system devices are listed as up (such as ge-0/0/16 and ge-0/0/17 for the Interconnect devices; ge-0/0/20 through ge-0/0/23 for the Director devices; ge-0/0/0 through ge-0/0/7 for the Node devices; and ge-0/1/0 and ge-0/1/1 for the inter-EX Series switch connections), the control plane is properly connected.

Verifying the QFX3000-M QFabric System Control Plane—EX Series Switch EX1

Purpose

Verify that the control plane is properly connected on your second EX Series switch.

Action

Connect to the Junos OS CLI of EX Series switch EX1, either from your management network or from the console port of the switch. In operational mode, enter the show interfaces terse command.

Sample Output
command-name

Meaning

In the output of the show interfaces terse command, if all interfaces that connect to the QFabric system devices are listed as up (such as ge-0/0/16 and ge-0/0/17 for the Interconnect devices; ge-0/0/20 through ge-0/0/23 for the Director devices; ge-0/0/0 through ge-0/0/7 for the Node devices; and ge-0/1/0 and ge-0/1/1 for the inter-EX Series switch connections), the control plane is properly connected.