Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

 
ON THIS PAGE
 

Migrate a Virtual Chassis Fabric to an EVPN-VXLAN Bridging Overlay Fabric

About This Network Configuration Example

This NCE shows how to migrate a four-member Virtual Chassis fabric (VCF) to an EVPN-VXLAN bridged overlay fabric. A VCF serves customers with a centrally managed Layer 2 fabric well. However, in a centrally managed fabric there can be downtime caused by maintenance and upgrades. You can avoid this downtime by moving to a distributed control plane. In this case, we recommend migrating your VCF to an EVPN-VXLAN bridged overlay fabric. Another reason to upgrade your VCF is that there will be limited support for VCF in future Junos releases.

Figure 1 shows a before and after of a VCF to EVPN-VXLAN configuration.

Figure 1: VCF to EVPN-VXLAN MigrationVCF network migration showing pre-migration VCF architecture with Master and Backup RE, QFX5100 switches, and post-migration VXLAN architecture with VTEP and ESI-LAG.

See Also

How to Migrate a Virtual Chassis Fabric to an EVPN Bridging Overlay Fabric

Requirements

We use the following in this example:

  • A two-spine and two-leaf VCF composed of QFX5100 switches running Junos OS Release 14.1X53-D47.6 that we will upgrade to Release 18.4R2.7, which is an EVPN-recommended release.

  • A server that is dual-homed to the VCF leaf devices. We recommend that the server be dual-homed to the leaf devices because a single homed server requires downtime to perform this migration.

  • Incorporate MTU 9216 on fabric (leaf and spine) underlay links and 9100 bytes on PE to CE links (AE interfaces level MTU).

  • Pre-provisioned mode VCF

  • Layer 2-only VCF

  • Console access to all devices

  • A reachable FTP server

  • Junos OS Release 18.4R2.7 or later EVPN recommended release

Overview

The key changes to move from a VCF to EVPN-VXLAN configuration are:

  • Adding an EBGP underlay

  • Adding an IBGP overlay

  • VLAN to VNI mapping

  • Addition of EVPN-VXLAN signaling, switch-options and related import and export policy statements

  • Changing LAG towards uplink device and and downlink servers to ESI-LAGs

To migrate the VCF to an EVPN-VXLAN fabric:

  1. Split the existing VCF into halves and isolate the backup Routing Engine and one device in a line card role.

  2. Migrate the isolated half to EVPN-VXLAN while traffic is still passing through the other half.

  3. Isolate and migrate the remaining half while directing the traffic through the new EVPN-VXLAN fabric.

  4. Join all the devices into a single EVPN-VXLAN fabric.

Topology

Figure 2 illustrates the topology of the VCF. Members 1 and 0 are connected to the uplink device, while the line cards are connected to the server.

Figure 2: Topology of the VCFNetwork topology diagram showing uplink device 192.168.100.1 connected to switches and server 192.168.100.100 with aggregated links for redundancy.

Configuration

Prepare for the Migration

Before you begin, create a topology diagram of your VCF like in Figure 2, copy the new Junos OS image to your devices, and monitor the traffic flow.

Step-by-Step Procedure

  1. Check the status of the VCF before you begin the migration. Note the serial numbers, member IDs, and associated roles of the devices.

  2. Check the Virtual Chassis ports (VCPs) and create a topology diagram for reference.

  3. Check that all four members are present. Check the Junos OS release running on each device. Each device must be running the same Junos OS release. If there is a release mismatch, the device shows as inactive.

  4. Copy the recommended Junos release for running EVPN to all the devices. Use FTP to copy the new Junos OS image to the primary Routing Engine. Then copy the new image from the primary Routing Engine to the other VCF members.

    To copy the image from the /var/tmp directory on the primary Routing Engine to Member 3, also called fpc 3:

    Figure 3 Illustrates how the new Junos OS image is distributed among the members.

    Figure 3: Copy Junos OS Image to VCF membersNetwork topology diagram showing Junos software upgrade process. Uplink device FTPs software to Master RE, with Backup RE as secondary. Members and line cards connected for traffic forwarding. Green arrows indicate traffic paths.

  5. Do the same for the other members. The FPC number is the same as the member number. For example:

  6. Access each member from the VCF primary Routing Engine and confirm that the file was copied to each member. For example, to access Member 3:

    Next, check the /var/tmp directory on this VCF member for the new Junos OS image.

    When you are done, use exit to return to the primary device.

    Repeat the image check on each device in the VCF.

  7. To check for any traffic loss during the procedure, start a continuous ping from the server to IRB 192.168.100.1 on the uplink device.

Reroute Traffic Through Member 1 and Member 3

To start the migration procedure, first isolate the half of the VCF that contains the backup Routing Engine and one device in the linecard role: Members 0 and 2. To do so, on Member 0, disable the interfaces to the uplink device and Member 3. On Member 2, disable the interface to Member 1 and the server. At this point, we have split the VCF in half. Traffic is still forwarded through the other line card, through the primary Routing Engine, and finally to the uplink MX Series router.

Figure 4: Isolate Member 0 and Member 2 Network topology diagram showing connections between devices. Uplink device connects to Master RE and Backup RE. S1 and S2 switches link to Master and Backup RE, with line cards L1 and L2 connected to switches. Server at bottom connects via bond0. Green line indicates traffic path. Red X marks show broken connections. Subnets 10.10.1.0/24 and 10.10.4.0/24 illustrated.
Step-by-Step Procedure

  1. Using the figure above, identify the member interfaces and VCPs you need to disable on Members 0 and 2 to isolate them from the rest of the VCF. The VCPs you disable are port 2 on Member 0 and port 53 on Member 2. Before you delete any VCPs enable ‘no split-detection’ on the entire VCF.

  2. Use the command below on the primary Routing Engine (Member 1) to determine the names of the relevant interfaces. You will disable the LACP member interfaces towards the uplink device and servers. In this case, et-0/0/23.0 is the Member 0 upstream interface and xe-2/0/46.0 is the Member 2 downstream interface.

  3. Access the primary (Member 1) console and do the following:

    • Disable the interface on Member 0 to the uplink device.

      Disable the interface from Member 2 to the server.

      Commit the configuration for it to take effect.

  4. Delete the VCP from Member 0 towards Member 3.

    • Delete the VCP from Member 2 towards Member 1.

  5. Check that the members were removed from the VCF and marked as NotPrsnt.

Upgrade Member 0 and Member 2

Separate member 0 and member 2 by deleting the VCP between both devices. On member 0 apply the following commands:

Now that you have isolated Member 0 and Member 2, you can upgrade these devices. Notice that the only traffic path is through Members 1 and 3.

Step-by-Step Procedure

  1. Access the consoles for Members 0 and 2. Enter the following command to upgrade both members to the Junos OS image that was copied onto the devices.

  2. Confirm the update was successful.

Zeroize Member 0 and Member 2

Once Member 0 and 2 are migrated to a recommended EVPN-VXLAN release, zeroize both switches. Zeroizing the switches ensures that the residual VCF configuration and logs are removed from each switch.

Once the switches reboot, restore the out-of-band management network connectivity.

Step-by-Step Procedure

Configure Member 0 and Member 2

Use the next steps to configure the underlay on Member 0 and Member 2. We identify the switches using member numbers for the sake of clarity even though the VCF does not exist yet. Configure the switches with the interface and loopback addresses, autonomous system (AS) numbers, and other components of the final EVPN-VXLAN fabric.

Figure 5: Final EVPN-VXLAN fabric Network topology diagram showing device connections, interfaces, IPs, and configurations. Uplink device IP 192.168.100.1, server IP 192.168.100.100, and member devices with AS numbers and loopbacks. Ethernet Segment Identifiers used for redundancy.
Step-by-Step Procedure

Configure the Underlay and EVPN-VXLAN Overlay for Member 0

Step-by-Step Procedure

  1. Configure the interfaces (Keep the uplink interface for Member 0 disabled).

  2. Configure the loopback interfaces for Member 0.

  3. Configure the underlay external BGP (EBGP) for Member 0.

  4. Configure the overlay internal BGP (IBGP) for Member 0.

  5. Configure the VLAN to VXLAN network identifier (VNI) mapping for Member 0.

  6. Configure the protocol EVPN-VXLAN configuration and route targets for each VNI for Member 0.

  7. Configure the default switch options for Member 0.

  8. Configure the routing options for Member 0.

  9. Configure the policy options for Member 0.

  10. On Member 0 configure ESI LAG to the uplink device. The ESI LAG pair for Member 0 is Member 1 after Member 1 is migrated to EVPN-VXLAN as seen in Figure 6.

    Figure 6: ESI LAG ConfigurationNetwork topology diagram with an uplink device at IP 192.168.100.1 connected to Member 1 AS 111 at subnet 10.1.0.0/24 and Member 0 AS 112 with subnets 10.10.2.0/24, 10.10.3.0/24, and 10.10.4.0/24 via aggregated Ethernet ae2 for redundancy or load balancing.

Configure the Underlay and EVPN-VXLAN Overlay on Member 2

Step-by-Step Procedure

  1. Configure the interfaces (keep the downlink interface for Member 2 disabled).

  2. Configure the loopback interfaces for Member 2.

  3. Configure the underlay EBGP for Member 2.

  4. Configure the overlay IBGP for Member 2.

  5. Configure the VLAN to VNI mapping for Member 2.

  6. Configure the protocol EVPN-VXLAN config and route targets for each VNI for Member 2.

  7. Configure the default switch options for Member 2.

  8. Configure the routing options for Member 2.

  9. Configure the policy options for Member 2.

  10. On Member 2, configure ESI LAG on the downlink to the server. The ESI LAG pair for Member 2 will be Member 3 after Member 3 is migrated to EVPN-VXLAN as seen in Figure 7.

    Figure 7: Configure the downlink ESI LAGNetwork topology diagram with two switches labeled Member 3 and Member 2, a server with IP 192.168.100.100, loopback addresses 10.1.1.1/32 (AS 113) and 10.1.1.2/32 (AS 114), subnets 10.1.0.0/24 and 10.10.4.0/24, and Ethernet Segment Identifier ESI 00:01:01:01:01:01:01:01:01:01.

Configure the Bond Interface on Host Server

In the output below from the server, eth4 and eth5 are both slave interfaces for bond0.

Move the Traffic Flow from VCF to the New EVPN-VXLAN Fabric

In this section, isolate Members 1 and 3 before configuring them for EVPN-VXLAN. Open the newly created EVPN-VXLAN fabric, Member 0 and Member 2, and reroute the traffic through it. Now that the EVPN-VXLAN configuration is in place on Member 0 and Member 2, check the BGP states between them. Member 1 and 3 still show as down since you haven’t configured EVPN-VXLAN on them yet.

Note:

NOTE: Shutting links on the Member 1 and 3 pair and opening the link on the Member 0 and 2 pair needs to be done simultaneously as seen in Figure 8 on page 31. You can do this using scripting.

Step-by-Step Procedure

Apply the following statements on the switches and commit them at the same time. It is important to follow these instructions.

Note:

Run commit at the same time on all devices. There might be a slight disruption in traffic until the states converge. Check that the traffic from the host passes successfully.

  1. On Member 1

  2. On Member 0

  3. On Member 2

Migrate and Zeroize Member 1 and Member 3

Separate member 1 and member 3 by deleting the VCP between both devices. On member1 CLI apply the following commands:

Step-by-Step Procedure

Now that Member 1 and 3 are isolated migrate both devices to recommended release for EVPN-VXLAN fabrics that we downloaded on all the switches at the beginning of this procedure.

  1. Migrate the switches.

  2. Zeroize the switches.

  3. Once the switches reboot, restore the out-of-band management network connectivity.

Configure the Underlay and EVPN-VXLAN Overlay for Member 1

Follow the same procedure as for Member 0 and Member 2.

Figure 8: Full Configuration of Member 1 and 3 Network topology diagram showing connections: Uplink device at IP 192.168.100.1 linked to aggregated Ethernet ae2, Members 0-3 with AS 112 to 113, Loopback IPs 10.1.1.12 to 10.1.1.1, and subnets 10.10.4.0 to 10.10.2.0. Server at IP 192.168.100.100 connected via ae1. Highlights redundancy and high availability.
Step-by-Step Procedure

Member 1 configuration:

  1. Configure the interfaces (keep the uplink interface for Member 1 disabled).

  2. Configure the loopbacks for Member 1.

  3. Configure the underlay EBGP for Member 1.

  4. Configure the overlay IBGP for Member 1.

  5. Configure the VLAN to VNI mapping for Member 1.

  6. Configure the protocol EVPN-VXLAN configurations and route targets for each VNI for Member 1.

  7. Configure the default switch options for Member 1.

  8. Configure the routing options for Member 1.

  9. Configure the policies for Member 1.

  10. On Member 1, configure ESI LAG to the uplink device. The ESI LAG pair for Member 1 will be Member 0 which we already configured in prior steps.

    Figure 9: Configure the ESI LAG going to Member 1Network topology diagram showing uplink device 192.168.100.1 connected to two switches with ESI 00:02:02:02:02:02:02:02:02:02. Member 1 AS 111 with loopback 10.1.1.11/32 connects to subnets 10.1.0.0/24 and 10.10.2.0/24. Member 0 AS 112 with loopback 10.1.1.12/32 connects to subnets 10.10.3.0/24 and 10.10.4.0/24. Connections via et-0/0/2 et-0/0/10 et-0/0/23 interfaces.

Configure the Underlay and EVPN-VXLAN Overlay for Member 3

Step-by-Step Procedure

Member 3 configuration:

  1. Configure the interfaces (keep the downlink interface for Member 3 disabled).

  2. Configure the loopbacks for Member 3.

  3. Configure the underlay EBGP for Member 3.

  4. Configure the overlay IBGP for Member 3.

  5. Configure the VLAN to VNI mapping for Member 3.

  6. Configure the protocol EVPN-VXLAN config and route targets for each VNI for Member 3.

  7. Configure the default switch options for Member 3.

  8. Configure the routing options for Member 3.

  9. Configure the routing options for Member 3.

  10. One Member 3, configure ESI LAG on the downlink to the host device. The ESI LAG pair for Member 3 will be Member 2 after Member 2 is migrated to to EVPN-VXLAN.

    Figure 10: Configure the Downlink ESI LAGNetwork topology diagram showing two devices, Member 3 AS 113 and Member 2 AS 114, with subnets 10.1.0.0/24 and 10.10.4.0/24, server IP 192.168.100.100, loopbacks 10.1.1.1/32 and 10.1.1.2/32, link aggregation with bond0 and ae1, and Ethernet Segment Identifier ESI 00:01:01:01:01:01:01:01:01:01.

Control Plane Convergence

At this step, we will open the links between all four member switches while keeping the uplink on Member 1 and the downlink on Member 3 disabled.

This is primarily for the convergence of BGP and EVPN-VXLAN states.

Figure 11: Open Links Between Switches Network topology diagram: Uplink device at 192.168.100.1 connects to switches via et-0/0/23 and et-0/0/10. Member 0 AS 112, Member 1 AS 111 with issues on et-0/0/23, Member 2 AS 114, and Member 3 AS 113 with issues on et-0/0/49. Server at 192.168.100.100 connects via bond0. Green checkmarks for active links; red X for issues.
Step-by-Step Procedure

  1. Check the BGP states.

  2. EVPN-VXLAN is MAC learning through the control plane. Once the control plane information has converged on Member 1 and Member 3, enable the disabled links, which are the uplink on Member 1 and downlink on Member 3. Perform the following steps and commit at the same time. Check for the traffic flow from the server to the uplink device.

    Figure 12: Open the Final Disabled Links Network topology diagram showing connections and traffic paths: Uplink device (192.168.100.1), four members with AS numbers and loopback IPs, server (192.168.100.100) via bond0, green arrows for traffic flow, and subnets 10.10.1.0/24 to 10.10.4.0/24.

    On Member 3

    On Member 1

Verification

At this point, the VCF fabric is migrated to an EVPN-VXLAN bridged overlay fabric. Perform the following check on every switch to confirm that the EVPN-VXLAN bridged overlay model is working as intended.

Step-by-Step Procedure

  1. Check that the ESI LAGs are up and running.

  2. Check that the Ethernet switching table has a MAC entry for the host.

  3. Check that the VLANs have VTEPS associated with them.

  4. Check the EVPN-VXLAN database is updated.

  5. Check that the routing table is receiving type 2 and other EVPN routes.

    Note:

    See Juniper Networks EVPN Implementation for Next-Generation Data Center Architectures for all EVPN route types.

Conclusion

This procedure outlines a step-by-step recommended method to migrate a centralized control plane VCF fabric to a distributed control plane Bridged overlay EVPN-VXLAN fabric.