Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

Migrate a Virtual Chassis Fabric to an EVPN-VXLAN Bridging Overlay Fabric

 

About This Network Configuration Example

This NCE shows how to migrate a four-member Virtual Chassis fabric (VCF) to an EVPN-VXLAN bridged overlay fabric. A VCF serves customers with a centrally managed Layer 2 fabric well. However, in a centrally managed fabric there can be downtime caused by maintenance and upgrades. You can avoid this downtime by moving to a distributed control plane. In this case, we recommend migrating your VCF to an EVPN-VXLAN bridged overlay fabric. Another reason to upgrade your VCF is that there will be limited support for VCF in future Junos releases.

Figure 1 shows a before and after of a VCF to EVPN-VXLAN configuration.

Figure 1: VCF to EVPN-VXLAN Migration
VCF to EVPN-VXLAN
Migration

How to Migrate a Virtual Chassis Fabric to an EVPN Bridging Overlay Fabric

Requirements

We use the following in this example:

  • A two-spine and two-leaf VCF composed of QFX5100 switches running Junos OS Release 14.1X53-D47.6 that we will upgrade to Release 18.4R2.7, which is an EVPN-recommended release.

  • A server that is dual-homed to the VCF leaf devices. We recommend that the server be dual-homed to the leaf devices because a single homed server requires downtime to perform this migration.

  • Incorporate MTU 9216 on fabric (leaf and spine) underlay links and 9100 bytes on PE to CE links (AE interfaces level MTU).

  • Pre-provisioned mode VCF

  • Layer 2-only VCF

  • Console access to all devices

  • A reachable FTP server

  • Junos OS Release 18.4R2.7 or later EVPN recommended release

Overview

The key changes to move from a VCF to EVPN-VXLAN configuration are:

  • Adding an EBGP underlay

  • Adding an IBGP overlay

  • VLAN to VNI mapping

  • Addition of EVPN-VXLAN signaling, switch-options and related import and export policy statements

  • Changing LAG towards uplink device and and downlink servers to ESI-LAGs

To migrate the VCF to an EVPN-VXLAN fabric:

  1. Split the existing VCF into halves and isolate the backup Routing Engine and one device in a line card role.
  2. Migrate the isolated half to EVPN-VXLAN while traffic is still passing through the other half.
  3. Isolate and migrate the remaining half while directing the traffic through the new EVPN-VXLAN fabric.
  4. Join all the devices into a single EVPN-VXLAN fabric.

Topology

Figure 2 illustrates the topology of the VCF. Members 1 and 0 are connected to the uplink device, while the line cards are connected to the server.

Figure 2: Topology of the VCF
Topology of the VCF

Configuration

Prepare for the Migration

Before you begin, create a topology diagram of your VCF like in Figure 2, copy the new Junos OS image to your devices, and monitor the traffic flow.

Step-by-Step Procedure

  1. Check the status of the VCF before you begin the migration. Note the serial numbers, member IDs, and associated roles of the devices.
  2. Check the Virtual Chassis ports (VCPs) and create a topology diagram for reference.
  3. Check that all four members are present. Check the Junos OS release running on each device. Each device must be running the same Junos OS release. If there is a release mismatch, the device shows as inactive.
  4. Copy the recommended Junos release for running EVPN to all the devices. Use FTP to copy the new Junos OS image to the master Routing Engine. Then copy the new image from the master Routing Engine to the other VCF members.

    To copy the image from the /var/tmp directory on the master Routing Engine to Member 3, also called fpc 3:

    Figure 3 Illustrates how the new Junos OS image is distributed among the members.

    Figure 3: Copy Junos OS Image to VCF members
    Copy Junos OS Image to VCF members
  5. Do the same for the other members. The FPC number is the same as the member number. For example:
  6. Access each member from the VCF master Routing Engine and confirm that the file was copied to each member. For example, to access Member 3:

    Next, check the /var/tmp directory on this VCF member for the new Junos OS image.

    When you are done, use exit to return to the master device.

    Repeat the image check on each device in the VCF.

  7. To check for any traffic loss during the procedure, start a continuous ping from the server to IRB 192.168.100.1 on the uplink device.

Reroute Traffic Through Member 1 and Member 3

To start the migration procedure, first isolate the half of the VCF that contains the backup Routing Engine and one device in the linecard role: Members 0 and 2. To do so, on Member 0, disable the interfaces to the uplink device and Member 3. On Member 2, disable the interface to Member 1 and the server. At this point, we have split the VCF in half. Traffic is still forwarded through the other line card, through the master Routing Engine, and finally to the uplink MX Series router.

Figure 4 shows the topology after this process is complete.

Figure 4: Isolate Member 0 and Member 2
Isolate Member 0 and Member 2

Step-by-Step Procedure

  1. Using the figure above, identify the member interfaces and VCPs you need to disable on Members 0 and 2 to isolate them from the rest of the VCF. The VCPs you disable are port 2 on Member 0 and port 53 on Member 2. Before you delete any VCPs enable ‘no split-detection’ on the entire VCF.
  2. Use the command below on the master Routing Engine (Member 1) to determine the names of the relevant interfaces. You will disable the LACP member interfaces towards the uplink device and servers. In this case, et-0/0/23.0 is the Member 0 upstream interface and xe-2/0/46.0 is the Member 2 downstream interface.
  3. Access the master (Member 1) console and do the following:

    • Disable the interface on Member 0 to the uplink device.

      Disable the interface from Member 2 to the server.

      Commit the configuration for it to take effect.

  4. Delete the VCP from Member 0 towards Member 3.

    • Delete the VCP from Member 2 towards Member 1.

  5. Check that the members were removed from the VCF and marked as NotPrsnt.

Upgrade Member 0 and Member 2

Separate member 0 and member 2 by deleting the VCP between both devices. On member 0 apply the following commands

Now that you have isolated Member 0 and Member 2, you can upgrade these devices. Notice that the only traffic path is through Members 1 and 3.

Step-by-Step Procedure

  1. Access the consoles for Members 0 and 2. Enter the following command to upgrade both members to the Junos OS image that was copied onto the devices.
  2. Confirm the update was successful.

Zeroize Member 0 and Member 2

Once Member 0 and 2 are migrated to a recommended EVPN-VXLAN release, zeroize both switches. Zeroizing the switches ensures that the residual VCF configuration and logs are removed from each switch.

Once the switches reboot, restore the out-of-band management network connectivity.

Configure Member 0 and Member 2

Use the next steps to configure the underlay on Member 0 and Member 2. We identify the switches using member numbers for the sake of clarity even though the VCF does not exist yet. Configure the switches with the interface and loopback addresses, autonomous system (AS) numbers, and other components of the final EVPN-VXLAN fabric as shown in Figure 5

Figure 5: Final EVPN-VXLAN fabric
Final EVPN-VXLAN fabric

Configure the Underlay and EVPN-VXLAN Overlay for Member 0

Step-by-Step Procedure

  1. Configure the interfaces (Keep the uplink interface for Member 0 disabled).
  2. Configure the loopback interfaces for Member 0.
  3. Configure the underlay external BGP (EBGP) for Member 0.
  4. Configure the overlay internal BGP (IBGP) for Member 0.
  5. Configure the VLAN to VXLAN network identifier (VNI) mapping for Member 0.
  6. Configure the protocol EVPN-VXLAN configuration and route targets for each VNI for Member 0.
  7. Configure the default switch options for Member 0.
  8. Configure the routing options for Member 0.
  9. Configure the policy options for Member 0.
  10. On Member 0 configure ESI LAG to the uplink device. The ESI LAG pair for Member 0 is Member 1 after Member 1 is migrated to EVPN-VXLAN as seen in Figure 6.
    Figure 6: ESI LAG Configuration
    ESI LAG Configuration

Configure the Underlay and EVPN-VXLAN Overlay on Member 2

Step-by-Step Procedure

  1. Configure the interfaces (keep the downlink interface for Member 2 disabled).
  2. Configure the loopback interfaces for Member 2.
  3. Configure the underlay EBGP for Member 2.
  4. Configure the overlay IBGP for Member 2.
  5. Configure the VLAN to VNI mapping for Member 2.
  6. Configure the protocol EVPN-VXLAN config and route targets for each VNI for Member 2.
  7. Configure the default switch options for Member 2.
  8. Configure the routing options for Member 2.
  9. Configure the policy options for Member 2.
  10. On Member 2, configure ESI LAG on the downlink to the server. The ESI LAG pair for Member 2 will be Member 3 after Member 3 is migrated to EVPN-VXLAN as seen in Figure 7.

Configure the Bond Interface on Host Server

In the output below from the server, eth4 and eth5 are both slave interfaces for bond0.

Move the Traffic Flow from VCF to the New EVPN-VXLAN Fabric

In this section, isolate Members 1 and 3 before configuring them for EVPN-VXLAN. Open the newly created EVPN-VXLAN fabric, Member 0 and Member 2, and reroute the traffic through it.

Now that the EVPN-VXLAN configuration is in place on Member 0 and Member 2, check the BGP states between them. Member 1 and 3 still show as down since you haven’t configured EVPN-VXLAN on them yet.

Note

Shutting links on the Member 1 and 3 pair and opening the link on the Member 0 and 2 pair needs to be done simultaneously as seen in Figure 8. You can do this using scripting.

Figure 8: Shutting and Opening Links
Shutting and Opening Links

Step-by-Step Procedure

Apply the following statements on the switches and commit them at the same time. It is important to follow these instructions.

Note

Run commit at the same time on all devices. There might be a slight disruption in traffic until the states converge. Check that the traffic from the host passes successfully.

  1. On Member 1
  2. On Member 0
  3. On Member 2

Migrate and Zeroize Member 1 and Member 3

Separate member 1 and member 3 by deleting the VCP between both devices. On member1 CLI apply the following commands

Step-by-Step Procedure

Now that Member 1 and 3 are isolated migrate both devices to recommended release for EVPN-VXLAN fabrics that we downloaded on all the switches at the beginning of this procedure.

  1. Migrate the switches.
  2. Zeroize the switches.
  3. Once the switches reboot, restore the out-of-band management network connectivity.

Configure the Underlay and EVPN-VXLAN Overlay for Member 1

Follow the same procedure as for Member 0 and Member 2. The full configuration for Member 1 and 3 is shown in Figure 9.

Figure 9: Full Configuration of Member 1 and 3
Full Configuration of Member 1 and
3

Step-by-Step Procedure

Member 1 configuration:

  1. Configure the interfaces (keep the uplink interface for Member 1 disabled).
  2. Configure the loopbacks for Member 1.
  3. Configure the underlay EBGP for Member 1.
  4. Configure the overlay IBGP for Member 1.
  5. Configure the VLAN to VNI mapping for Member 1.
  6. Configure the protocol EVPN-VXLAN configurations and route targets for each VNI for Member 1.
  7. Configure the default switch options for Member 1.
  8. Configure the routing options for Member 1.
  9. Configure the policies for Member 1.
  10. On Member 1, configure ESI LAG to the uplink device. The ESI LAG pair for Member 1 will be Member 0 which we already configured in prior steps.

Configure the Underlay and EVPN-VXLAN Overlay for Member 3

Step-by-Step Procedure

Member 3 configuration:

  1. Configure the interfaces (keep the downlink interface for Member 3 disabled).
  2. Configure the loopbacks for Member 3.
  3. Configure the underlay EBGP for Member 3.
  4. Configure the overlay IBGP for Member 3.
  5. Configure the VLAN to VNI mapping for Member 3.
  6. Configure the protocol EVPN-VXLAN config and route targets for each VNI for Member 3.
  7. Configure the default switch options for Member 3.
  8. Configure the routing options for Member 3.
  9. Configure the routing options for Member 3.
  10. One Member 3, configure ESI LAG on the downlink to the host device. The ESI LAG pair for Member 3 will be Member 2 after Member 2 is migrated to to EVPN-VXLAN.
    Figure 11: Configure the Downlink ESI LAG
    Configure the Downlink ESI LAG

Control Plane Convergence

At this step, we will open the links between all four member switches while keeping the uplink on Member 1 and the downlink on Member 3 disabled as seen in Figure 12. This is primarily for the convergence of BGP and EVPN-VXLAN states.

Figure 12: Open Links Between Switches
Open Links Between Switches

Step-by-Step Procedure

  1. Check the BGP states.
  2. EVPN-VXLAN is MAC learning through the control plane. Once the control plane information has converged on Member 1 and Member 3, enable the disabled links, which are the uplink on Member 1 and downlink on Member 3. Perform the following steps and commit at the same time. Check for the traffic flow from the server to the uplink device.
    Figure 13: Open the Final Disabled Links
    Open the Final Disabled Links

    On Member 3

    On Member 1

Verification

At this point, the VCF fabric is migrated to an EVPN-VXLAN bridged overlay fabric. Perform the following check on every switch to confirm that the EVPN-VXLAN bridged overlay model is working as intended.

Step-by-Step Procedure

  1. Check that the ESI LAGs are up and running.
  2. Check that the Ethernet switching table has a MAC entry for the host.
  3. Check that the VLANs have VTEPS associated with them.
  4. Check the EVPN-VXLAN database is updated.
  5. Check that the routing table is receiving type 2 and other EVPN routes.Note

    See Juniper Networks EVPN Implementation for Next-Generation Data Center Architectures    for all EVPN route types.

Conclusion

This procedure outlines a step-by-step recommended method to migrate a centralized control plane VCF fabric to a distributed control plane Bridged overlay EVPN-VXLAN fabric.