Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

 
ON THIS PAGE
 

EVPN Maintenance Mode for Multihomed Leaf Isolation

EVPN Maintenance Mode enables seamless maintenance and upgrades in EVPN environments by isolating multihomed Leaf nodes, rerouting traffic to maintain network stability.

EVPN maintenance mode streamlines maintenance and upgrade operations in EVPN environments by minimizing traffic disruption. By isolating devices in a controlled manner, service interruptions during upgrades can be prevented. This feature operates seamlessly with multihomed leaf nodes, rerouting traffic to alternate nodes to maintain network stability. It includes pre-checks and validations to ensure multihomed configurations, service tracking, and non-revertive designated forwarder (DF) settings are correctly configured. Non-revertive DF and egress link protection (ELP) enhance resilience, prevent traffic loss and improve convergence times, while network isolation (NISO) profiles aid in maintaining service continuity through core isolation management. You use the command-line interface (CLI) commands to configure, validate, and monitor maintenance mode, and to ensure that all pre-check requirements are met before network changes.

Benefits of Maintenance Mode CLI

  • Minimizes network disruption by isolating multihomed leaf nodes during upgrades, ensuring continued network operations.

  • Enhances system resilience with non-revertive DF election, maintaining stability even during node outages.

  • Improves route convergence times through ELP, ensuring efficient traffic rerouting and reduced downtime.

  • Supports multihoming configurations, providing robust connectivity options that maintain network performance during maintenance activities.

  • Ensures compatibility and optimal performance by adhering to recommended pre-check and post-check procedures, reducing the risk of upgrade failures.

  • Facilitates service continuity and stability during maintenance using NISO profiles to manage core isolation and interface actions effectively.

Overview

EVPN maintenance mode is a sophisticated feature that facilitates seamless maintenance and upgrades in EVPN networks by isolating multihomed leaf nodes and shutting down access-side interfaces. Activating this mode minimizes traffic disruption by ensuring effective traffic rerouting, maintaining network performance during maintenance and system upgrades. You initiate maintenance mode using the CLI command set protocols evpn maintenance-mode deployment-mode action-type action-type. The options validate-only, validate-and-set, or force give you precise control over the process, ensuring that pre-checks and configurations are accurate before proceeding.

You can validate configurations before and after starting maintenance mode with the request evpn validate-maintenance-mode-pre-checks command. Then use show evpn maintenance-mode-status to view results of the pre-checks and monitor the current status. If you change any maintenance mode related configurations, rerun request evpn validate-maintenance-mode-pre-checks to update the validation.

The egress link protection (ELP) and fast reroute (FRR) features are integral to maintaining network reliability, especially during link failures. When a link to a customer edge device fails, these features facilitate rapid route convergence, rerouting traffic swiftly to avoid significant data loss. This proactive approach ensures that during maintenance or unexpected failures, the network remains stable and operational. The non-revertive DF election further stabilizes the network by maintaining consistent designated forwarder status even after node reboots, preventing unnecessary traffic switching and ensuring a seamless transition back to normal operations. Network isolation (NISO) profiles also play a crucial role in core isolation management, vital during maintenance activities. These profiles manage interface actions and service-tracking mechanisms to ensure continuity and stability.

Starting in Junos OS and Junos OS Evolved Release 26.2R1, EVPN maintenance mode supports collapsed core, core distribution, edge-routed bridging (ERB), and IP Clos deployments. Users can schedule the start and optional end of maintenance mode, validate NTP-based DF election for scheduled maintenance, and use enhanced status output to monitor validation and timer states. Maintenance mode also withdraws locally originated EVPN Type 5 host and subnet routes when scheduled maintenance begins and supports data center interconnect (DCI) gateway prechecks and recovery delay behavior.

The maintenance mode CLI feature was introduced in Junos OS release 25.3R1 for ERB deployments. Junos OS 26.2R1 added support for Collapsed Core, Core Distribution, IP Clos deployments and scheduling options. Please refer to Feature Explorer EVPN maintenance mode for a complete list of the products that support this feature.

EVPN Maintenance Mode Prerequisites

Before initiating EVPN maintenance mode, prepare your network by ensuring out-of-band management capabilities and bandwidth availability to handle potential traffic loads when traffic is diverted to the multihomed peer. This feature does not support single-homed configurations or in-band network management during maintenance.

Adhering to operational guidelines is crucial; these include conducting pre-maintenance health checks, backing up configurations and cleaning system storage. After completing the maintenance activity verify the system and network status to confirm stability and performance.

The maintenance mode CLI feature has the following prerequisites for implementation.

  • Multihomed EVPN instances with the Ethernet segment identifier (ESI) configured with df-election-type preference non-revertive.

  • NISO profiles with service-tracking core-isolation and service-tracking-action link-down configurations.

  • Access interfaces configured with hold-time up for better traffic convergence post maintenance.

  • (EVPN-MPLS) ELP (evpn-egress-link-protection) configured under the routing-options forwarding-table hierarchy.

  • (EVPN-VXLAN) FRR (reroute-address) configured under the forwarding-options evpn-vxlan hierarchy.

  • Minimum software version of Junos OS 25.3R1 or later for ERB deployments.

  • Minimum software version of Junos OS 26.2R1 or later for Collapsed Core, Core Distribution, IP Clos deployments and scheduling options.

  • For a DCI gateway, configure preference-based non-revertive DF election for the interconnect Ethernet segment identifier (I-ESI).

  • For scheduled maintenance, configure NTP-based DF election:

    set protocols evpn df-election-ntp

Note:

When you schedule maintenance, the pre-check validates the NTP-based DF election configuration in addition to the existing ELP or FRR, NISO profile, and non-revertive DF checks.
Note: You manage EVPN maintenance mode using configuration statements. When you exit maintenance mode by removing the maintenance-mode configurations the hold-time configured in the NISO groups is not triggered. Therefore, the hold-time up should be configured on the access interfaces.
CAUTION:

For campus-fabric core nodes, EVPN maintenance mode does not automatically drain WAN or other external traffic. Schedule external BGP graceful shutdown or hard shutdown and configure OSPF or IS-IS overload as required for the deployment.
Note:

On EX4100, EX9204, and EX9208 platforms, the forwarding-options evpn-vxlan reroute-address statement is not supported. A default platform parameter skips the ELP or FRR reroute-address pre-check on these platforms. Because reroute-address is unavailable, access-link failure during maintenance can cause additional traffic loss.

Non-revertive DF

The non-revertive option for Preference-Based DF Election ensures that once a DF is elected, it will not be preempted by the previously designated DF coming back online after a failure. This non-revertive mode is key in maintaining a stable network environment and avoiding unnecessary service disruptions.

Network Isolation Groups and Service Tracking

Network isolation (NISO) groups define what to track to detect core isolation and what action to take on layer 2 interfaces. Core isolation tracking can be based on core side link tracking or service tracking.

You define NISO groups using the set protocols network-isolation group statement. Then you associate the group name with CE facing interfaces using the network-isolation-profile network-isolation-group-name statement under the edit interfaces interface name hierarchy. This configures those interfaces for fast switchover of traffic in event of core isolation.

The maintenance mode CLI disables access-facing interfaces by sending an EVPN maintenance-mode message to the Layer 2 Address Learning Daemon (L2ALD). This causes the interfaces connected to the multihomed CE to go into an LACP out-of-sync (OOS) state. However, the LACP interfaces may not go down right away. So, you must configure the service-tracking core-isolation and service-tracking-action link-down statements in the NISO group to bring down the maintenance node's interfaces immediately, allowing better and faster traffic convergence. Additionally, setting the interfaces name hold-time down to 0 ensures the interface shuts down immediately when the Maintenance Mode CLI is enabled.

You can use the interfaces name hold-time up statement to configure a short delay when bringing the access facing interfaces back up. This improves traffic convergence by enabling the core facing interfaces to come up and synchronize routes with the neighboring PEs first. The hold-time configured in NISO group is not triggered when removing the maintenance mode CLI configuration.

ELP and FRR Configurations

You configure the evpn-egress-link-protection statement for EVPN-MPLS or the reroute-address statement for EVPN-VXLAN to enable the egress link protection fast reroute (ELP/FRR) feature on multihoming peer provider edge (PE) devices. This feature helps to minimize load-balanced traffic loss when the link from a PE device to a multihomed CE device goes down.

The ELP/FRR check passes when there is a valid EVPN-MPLS or EVPN-VXLAN style configuration as shown below.

EVPN-MPLS configuration:

EVPN-VXLAN configuration:

Command Utilization and Best Practices

To effectively implement the maintenance mode CLI, it is essential to understand the commands and their usage.

The request evpn validate-maintenance-mode-pre-checks deployment-mode command verifies that the device configuration meets the maintenance-mode prerequisites. This command initiates the same validation checks as the set protocols evpn maintenance-mode deployment-mode action-type action-type configuration and can be ran before, during or after configuring maintenance-mode. You should run this command after any configuration changes to ensure those changes are considered during re-validation. Then use the show evpn maintenance-mode-status command to check the status.

The show evpn maintenance-mode-status displays the results of the most recent validation checks and the whether the current EVPN Maintenance Status: status is Under Maintenance or Not-under Maintenance.

  • Under Maintenance—Indicates the access facing interfaces are disabled/down and all EVPN T2 and T4 routes have been withdrawn and the node is ready for maintenance or upgrade.

  • Not-under Maintenance—Indicates the node is not in maintenance-mode. Use the set protocols evpn maintenance-mode deployment-mode action-type command with the validate-and-set or force option to enter maintenance-mode.

You should regularly consult show evpn maintenance-mode-status to assess the current mode and review validation results. These insights are crucial for making informed decisions during maintenance activities.

show evpn maintenance-mode-status

Use the show evpn maintenance-mode-status extensive command to view the overall precheck result, individual NTP-based DF result, configured maintenance times, maintenance action times, and start and end timer states. This output also displays untracked interfaces. These are non-EVPN interfaces that might go down during the maintenance if they have a NISO profile configured.

For DCI gateways, the extensive output also displays the I-ESI non-revertive DF validation result.

show evpn maintenance-mode-status extensive

Configure maintenance mode for the applicable deployment type:

Where:

  • deployment-mode is collapsed-core, core-distribution, erb-leaf, or ip-clos.

  • action-type is validate-only, validate-and-set, or force.

  • schedule-start-at schedules maintenance to begin at the specified local system time.

  • schedule-end-at optionally schedules the end of maintenance mode. The end time must be later than the start time.

  • time format is: [yyyy-mm-dd.]hh:mm

Do not configure schedule-start-at or schedule-end-at with the validate-only action type. The validate-only action type performs pre-checks only and does not enter maintenance mode.

Use the force action type only when you understand the consequences of bypassing failed pre-checks. The device still performs pre-checks, but it ignores their results. If no schedule is configured, the device enters maintenance mode after the pre-check job completes. If a start time is configured, the device enters maintenance mode at the scheduled time.

If you delete or modify the maintenance-mode configuration, the device cancels existing schedule timers and creates new timers as needed. If the configured end time is in the past, the device does not start maintenance actions. If the start time is in the past and pre-checks pass, the device starts maintenance immediately; it does not perform the scheduled DF-preference adjustment.

The validation options certify that your network is adequately configured before entering maintenance mode. These pre-checks perform the following checks:

  • validate multihomed configurations

  • enforce service tracking actions

  • ensure non-revertive DF settings

  • verify NTP-based DF election for scheduled maintenance

By ensuring these conditions are met, the risk of service disruption during maintenance is reduced. Additionally, utilizing non-revertive DF election and ELP with fast reroute (FRR) enhances network convergence times and resilience, preventing traffic loss during maintenance operations. NISO profiles play a crucial role in core isolation management, vital during maintenance activities. These profiles manage interface actions and service-tracking mechanisms to ensure continuity and stability.

CAUTION:
The maintenance mode CLI feature assumes that the configuration includes NISO profiles attached to all CE facing L2 interfaces and not attached to any core facing interfaces. It does not validate whether interfaces with NISO profiles are CE facing or core facing. Incorrectly configured NISO profiles can cause traffic loss.
Note: The set protocols evpn maintenance-mode and the set protocols l2-learning niso-maintenance down commands cannot be configured at the same time.

Use the delete protocols evpn maintenance-mode command to remove the maintenance-mode configuration and bring the EVPN instances back online and resume normal operations. You can implement this once maintenance is completed to restart EVPN immediately. Or you can leave the maintenance-mode configured and remove it after a reboot if necessary.

For example, if you reboot the node and maintenance-mode is configured with the force option all the CE facing interfaces would remain down. If maintenance-mode is configured with the validate-and-set option it would validate the pre-checks and act accordingly.

Alternatively, if you change the maintenance-mode to use the validate-only option it will clear the maintenance-mode and bring the CE facing interfaces up.

Scheduled Maintenance Processing

When the scheduled start timer expires, the device verifies that the current maintenance pre-check status is PASS. If validation fails, the device does not enter maintenance mode and logs EVPN_MAINTENANCE_MODE_PRECHECK_FAILED.

When validation passes, the device performs the following actions:

  • Changes the active DF preference and no-preempt settings, with jitter, for locally hosted ESIs so that the device relinquishes DF responsibility.

  • Withdraws locally originated EVPN Type 5 host and subnet routes. The device retains the lo0 prefix so that remote PEs retain the de-encapsulation route.

  • Starts a 60 second delay timer before entering maintenance mode.

    Note: During the 60-second delay, the system expects all locally hosted ESIs to relinquish DF status. If transient network events cause the maintenance node to remain the DF for one or more ESIs, maintenance can cause additional traffic loss.

  • Sends the maintenance-mode-detected action to bring down access interfaces that have an enabled NISO profile.

Note:

If a configuration change causes pre-checks to fail after maintenance mode has started, the device immediately clears maintenance mode. Otherwise, the device automatically exits maintenance mode at the configured schedule-end-at time. To exit earlier, delete the maintenance-mode configuration.
Note:

The schedule-end-at time configuration is optional. If you do not configure a schedule-end-at time, you exit maintenance mode by deleting the maintenance-mode configuration.
CAUTION:

The device does not re-validate whether another multihoming peer is available immediately before it changes DF preference or brings down access interfaces. Make sure that every locally hosted ESI has a reachable multihoming peer before scheduled maintenance begins. A peer failure or another transient network event can cause traffic loss if the maintenance node remains the DF for an ESI when its access interfaces are brought down.

EVPN Type 5 Route Behavior

At the scheduled maintenance start time, the device withdraws locally originated EVPN Type 5 host and subnet routes and suppresses new Type 5 advertisements while maintenance mode is active. When maintenance mode ends or is cleared, the device resumes Type 5 advertisements after a delay that allows multihomed access links to come up and local MAC routes to be programmed.

EVPN maintenance mode does not provide ELP or FRR protection for EVPN Type 5 traffic. Withdrawing Type 5 routes can result in traffic loss during maintenance.

DCI Gateway Behavior

For scheduled maintenance on a DCI gateway, the device withdraws I-ESI Ethernet Auto-Discovery (A-D) and Ethernet segment routes for EVPN Type 2 traffic. To delay recovery of all I-ESIs after maintenance, configure:

set protocols evpn interconnect-esi-bringup-delay seconds

The range is 1 through 600 seconds. This statement overrides the default 10-second I-ESI bringup timer.

For EVPN Type 5 traffic, DCI gateway behavior is limited to the Type 5 route actions described in EVPN Type 5 Route Behavior.

Upgrade with Maintenance Mode

The various deployment types provide layer 2 (L2) and layer 3 (L3) VXLAN functionality differently. But they each use multihomed access interfaces to provide connectivity to attached workloads. You can use the maintenance mode CLI to disable and enable the end-device facing access interfaces using NISO profiles to minimize network disruption during maintenance activities. The schedule-start-at and schedule-end-at options enable you to schedule the maintenance actions for a future time.

You define NISO groups and attach them to layer 2 or CE facing interfaces for fast switchover of traffic in the event of core isolation. These groups define what to track to detect core isolation and what action to take on layer 2 interfaces. You can use the maintenance mode CLI to leverage the NISO groups during maintenance or upgrades to simplify isolating multihomed nodes and minimize traffic disruption.

The Junos OS Upgrade for EVPN VXLAN Network document provides detailed recommended procedures for upgrading EVPN-VXLAN deployments. Chapter 2 covers the guidelines for planning the upgrade and includes the recommended pre-upgrade and post-upgrade health checks. Chapter 3 provides an overview and specific steps for upgrading an ERB leaf node. Chapters 4, 5, and 6 cover centrally routed bridging (CRB) deployments. You can use the maintenance mode CLI to isolate the multihomed access interfaces for the steps that disable and enable the end-device facing access interfaces.

Enter EVPN Maintenance Mode

EVPN maintenance mode leverages the NISO configurations to disable the end-device facing access interfaces and divert traffic towards multihomed peer devices. This helps to isolate the node and prepare it for maintenance.

Use the following steps to enable maintenance mode.

  • Use the request evpn validate-maintenance-mode-pre-checks deployment-mode and show evpn maintenance-mode-status commands to ensure the device configuration and state meets the prerequisites for EVPN maintenance mode.

  • Enter maintenance mode by configuring the set protocols evpn maintenance-mode deployment-mode action-type action-type statement with either the validate-and-set or the force option.

    The action-type options are:

    • validate-only—initiate the EVPN maintenance-mode pre-check process and provide the validation results.

    • validate-and-set—initiate the EVPN maintenance-mode pre-checks and if the validations pass then enter maintenance-mode.

    • force—skip the EVPN maintenance-mode pre-checks and enter maintenance-mode immediately.

Exit EVPN Maintenance Mode

You exit maintenance mode to enable the end-device facing access interfaces that maintenance mode disabled. The maintenance mode CLI configurations remain active as long as they are configured, even after a reboot. For example:

  • If you configure maintenance-mode with the validate-and-set option, it will validate the pre-checks after rebooting and act accordingly.

  • If you configure maintenance-mode with the force option, it will skip the pre-checks after rebooting and re-enter maintenance-mode immediately.

You can exit maintenance-mode by changing the maintenance-mode option to validate-only or by removing the maintenance-mode CLI configuration as follows:

  • Use set protocols evpn maintenance-mode deployment-mode action-type validate-only to exit maintenance-mode but still run the validation checks.

  • Use the delete protocols evpn maintenance-mode command to exit maintenance-mode by removing the configuration.

Note: When you exit Maintenance Mode the hold-time configured in the NISO groups is not triggered. Therefore, the hold-time up should be configured on the access interfaces.

Schedule EVPN Maintenance Mode

Use the schedule-start-at and schedule-end-at options to schedule maintenance for a future time.

Before the scheduled start time, verify that the maintenance pre-checks pass. At the scheduled start time, the device relinquishes DF responsibility and withdraws locally originated EVPN Type 5 host and subnet routes. The device starts the NISO interface action 60 seconds later.

At the scheduled end time, the device clears the maintenance mode state. The schedule-end-at time configuration is optional. If you do not configure a schedule-end-at time, you exit maintenance mode by deleting the maintenance-mode configuration.

Do not configure schedule-start-at or schedule-end-at with the validate-only action type. The validate-only action type performs pre-checks only and does not enter maintenance mode.

Related Documentation