Test Objectives

Test Non-Goals

Maximum scale and performance of the individual network elements constitute the solution. There is no preferred specification for the hypervisor hosting the vSRX firewall, nor any specific vSRX sizes (in vCPU/vRAM/vNIC quantity). Simple vSRX firewalls are enough for testing the features. Note that vSRX firewall runs on many hypervisors including: ESXi, KVM, and Microsoft for onprem. Though vSRX firewall can also be deployed in public clouds like AWS, Azure and GCP, the purpose of the architecture is not to run with vSRX firewall in those external clouds where it may be questionable to consider the networking plumbing to get them connected.

Following features and functions are not included in this JVD:

• Automated onboarding of the vSRX firewall
• Security director
• Network Address Translation: NAT64, DetNAT, PBA, DS-Lite
• Load balancing: filter-based forwarding
• Application and Advanced Security features like AppID, IDP, URL filtering, and another Layer 7

Tested Failure Events

SRX Series Firewalls failure events:

• MX Series Router to SRX Series Firewall link failures
• SRX Series Firewall reboot
• SRX Series Firewall power off
• Complete MNHA pair power off

MX failure events:

• Reboot MX Series Router
• Restart routing process
• Restart TLB process in MX router (traffic-dird, sdk-process and netmon deamon)
• GRES (Graceful Restart of routing daemon)
• ECMP/TLB next-hop addition or deletion (adding or deleting a new scale-out SRX MNHA pair)
• SRD based CLI switchover between MX Series Router (ECMP)

Traffic recovery is validated post all failure scenarios.

UDP traffic generated using IxNetwork for all the failure related test cases is used to measure the failover convergence time.

Tested Traffic Profiles

Tested traffic profiles are composed of multiple simultaneous flows for either a standalone SRX Series Firewall or a SRX MNHA pair in Active/Backup mode.

Packet size is using Internet mix with average packet size of ~700bytes. The Packet Size:Weight distribution is as follows:

• 64:8
• 127:36
• 255:11
• 511:4
• 1024:2
• 1518:39

Test Bed Configuration

Contact your Juniper representative to obtain the full archive of the test bed configuration used for this JVD.

Traffic Path in SFW and CGNAT Scale-Out Solution

The scale-out solution is based on BGP as dynamic routing protocol. It enables all the MX Series Router and SRX Series Firewalls to learn of their surrounding networks, however, most importantly to exchange path information for the network traffic that needs to be sent from the MX Series Router across each SRX Series Firewalls to the next MX Series Router. This protocol enables the exchange of network paths for the internal/user subnets and the default/specific external network. When each SRX Series Firewalls announces what it learned from the other side, each with the same “network cost”, the load balancing can then use those routes for load balancing traffic across each SRX Series Firewalls.

Figure 1: BGP Network Announcements

The following diagram shows how traffic flows may be distributed from an MX Series Router to multiple SRX Series Firewalls using ECMP load balancing method. The SRX Series Firewalls are in a symmetric sandwich between the two MX Series Routers in the diagram, whether those MX routers are actually a single physical node configured with two routing instances (more typical) or two physical MX Series Router nodes on each side, the routing principle stays the same as if two routing nodes are used, maintaining the traffic flow distribution that is consistent in both directions.

Figure 2: Traffic Flows

The MX Series Router on the left uses TRUST-VR routing instance to forward traffic to each SRX Series Firewall.

The MX Series Router on the right has used UNTRUST-VR to receive traffic from each SRX Series Firewall and forward it to the next-hop toward the target resources. The routes on each side are announced through BGP to the next hop, making its path available on each MX instance through each SRX Series Firewall (with same cost for load balancing).

Routes are announced through BGP, each MX router with their own BGP Autonomous System (AS) and peer with the SRX Series Firewall on their two sides (TRUST and UNTRUST zones in a single routing instance). The MX Series Router may peer with any other routers bringing connectivity to the clients and servers (here GW Router).

When the routes across each SRX Series Firewalls are known with similar cost, then the load balancing method can be used as explained below.

For CGNAT use case, this is very similar to SFW, however the NAT pools are exchanged on the right MX Series Router for the return traffic to flow back to the correct SRX Series Firewall:

Figure 3: Network BGP Announces with NAT Pools

Introduction to SRX Multi Node High Availability

For more information, see an extract from the public documentation on MNHA https://www.juniper.net/documentation/us/en/software/junos/high-availability/topics/topic-map/mnha-introduction.html.

Juniper Networks® SRX Series Firewalls support a new solution, Multi Node High Availability (MNHA), to address high availability requirements for modern data centres. In this solution, both the control plane and the data plane of the participating devices (nodes) are active at the same time. Thus, the solution provides inter-chassis resiliency.

The participating devices are either co-located or physically separated across geographical areas or other locations such as different rooms or buildings. Having nodes with HA across geographical locations ensures resilient service. If a disaster affects one physical location, MNHA can fail over to a node in another physical location, thereby ensuring continuity.

In MNHA, both SRX series firewalls have an active control plane and communicate their status over an Inter Chassis Link (ICL) that can be direct or routed across the network. This allows the nodes to be geo-dispersed while synchronizing the sessions and IKE security associations. Also, they do not share a common configuration, and this enables different IP addresses settings on both SRX series firewalls. There is a commit sync mechanism that can be used for the elements of configuration to be same on both platforms.

The SRXs uses one or more SRDs for the data plane that can be either active or backup (for SRG1 and above). An exception is the SRG group 0 (zero) that is always active on both. This is a group that can be used natively by scale-out solution to load balance the traffic across both SRX Series Firewalls at the same time. However, some interest exists for the other modes where it can be Active/Backup for SRG1 and Backup/Active for SRG2. This is like always active SRG0, however can also add some routing information (like BGP as-path-prepend) under certain conditions. SRG1/+ offers more health checking of its surrounding environment that can be leveraged to make an SRGn group active/backup/ineligible.

Figure 4: Munti Node High Availability General Architecture

MNHA can select a network mode between the following three possibilities:

• Default Gateway or L2 mode: It uses only the same network segment at L2 on the different sides of the SRX Series Firewalls (e.g. trust/untrust) and both SRX Series Firewalls share a common IP / MAC address on each network segment. It does not mean the SRX Series Firewall is in switching mode, it does route between its interfaces, however, shares the same broadcast domain on one side with the other SRX Series Firewall, and same on the other side as well.
• Hybrid mode or mix of L2 and L3: It uses an L2 and IP address on one side of the SRX Series Firewall (e.g. trust) and routing on the other side (e.g. untrust) then having different IP subnets on the second side.
• Routing mode or L3: This is the architecture used for this JVD where each side of the SRX Series Firewall is using different IP address, even between the SRX Series Firewalls (no common IP subnet) and all communication with the rest of the network is done through routing. This mode is perfect for scale-out communication using BGP with the MX Series Router.

Figure 5: Multi Node High Availability Network Modes

Whether using SRG0 Active/Active, or SRG1 Active/Backup (single one active at a time), or a combination of SRG1 Active/Backup and SRG2 Backup/Active, this simply uses one or two SRX Series Firewalls in a cluster at the same time.

ECMP/Consistent Hashing (CHASH) Load Balancing Overview

This feature relates to topology 1 (single MX Series Router, scale-out SRXs) and topology 2 (dual A/P MX Series Router and scale-out SRX MNHA pairs).

Figure 6: : Topologies 1 and 2 - ECMP CHASH

ECMP/Consistent Hashing (CHASH) in MX Router:

ECMP is a network routing strategy that transmits traffic of the same session, or flow — that is, traffic with the same source and destination across multiple paths of equal cost. It is a mechanism that allows to load balance traffic and increase bandwidth (by fully utilizing) otherwise unused bandwidth on links to the same destination.

When forwarding a packet, the routing technology must decide which next-hop path to use. The device considers the packet header fields that identify a flow. When ECMP is used, next-hop paths of equal cost are identified based on routing metric calculations and hash algorithms. That is, routes of equal cost have the same preference and metric values, and the same cost to the network. The ECMP process identifies a set of routers, each of which is a legitimate equal cost next-hop towards the destination. The routes that are identified are referred to as an ECMP set. An ECMP set is formed when the routing table contains multiple next-hop addresses for the same destination with equal cost (routes of equal cost have same preference and metric values). If there is an ECMP set for the active route, Junos OS uses a hash algorithm to choose one of the next-hop addresses in the ECMP set to install in the forwarding table. You can configure Junos OS so that multiple next-hop entries in an ECMP set are installed in the forwarding table. On Juniper Networks devices, per-packet load balancing is performed to spread traffic across multiple paths between routing devices.

The following example is of learned routes and forwarding table for the same destination (assuming traffic target is within 100.64.0.0/16 and SRX BGP peers are 10.1.1.0, 10.1.1.8 and 10.1.1.16):

With scale-out architecture where stateful security devices are connected, maintaining symmetricity of the flows in the security devices is the primary objective. The symmetricity means traffic from a subscriber (user) and to the subscriber must always reach the same server (which maintains the subscriber state). To reach the same server, the traffic must be hashed onto the same link towards that server for traffic in both directions.

A subscriber is identified by the source IP address in the upstream direction (client to server) and by the destination IP address in the downstream direction (server to client). The MX Series Routers do symmetric hashing i.e. for a given (sip, dip) tuple, same hash is calculated irrespective of the direction of the flow i.e. even if sip and dip are swapped. However, the requirement is that all flows from a subscriber reach the same SRX Series Firewall so you need to hash only on source IP address (and not destination IP address) in one direction and vice versa in the reverse direction.

By default, when a failure occurs in one or more paths, the hashing algorithm recalculates the next hop for all paths, typically resulting in redistribution of all flows. Consistent load balancing enables you to override this behavior so that only flows for inactive links are redirected. All existing active flows are maintained without disruption. In such an environment, the redistribution of all flows when a link fails potentially results in significant traffic loss or a loss of service to SRX Series Firewall whose links remain active. However, consistent load balancing maintains all active links and remaps only those flows affected by one or more link failures. This feature ensures that flows connected to links that remain active continue to remain uninterrupted.

This feature applies to topologies where members of an ECMP group are external BGP neighbors in a single-hop BGP session. Consistent load balancing does not apply when you add a new ECMP path or modify an existing path in any way. New SRX add design is implemented recently where you can add SRX Series Firewall gracefully with an intent of equal redistribution from each active SRX Series Firewall, hence causing minimal impact to the existing ECMP flows. For example, if there are four active SRX Series Firewalls carrying 25% of total flows on each link and a 5th SRX Series Firewalls (previously unseen) is added, 5% of flows from each existing SRX Series Firewalls moves to the new SRX Series Firewalls. Hence making 20% of flow re-distribution from existing four SRX Series Firewalls to the new one.

The following information shares details for each step of route exchange between MX Series Router and SRXs, traffic flows, for each use case.

ECMP/CHASH in Topology 1 (Single MX, scale-out SRXs) for SFW:

Figure 7: Topology 1 - ECMP CHASH - SFW Use Case

• SRX Series Firewalls are deployed in a standalone scaled out devices to single MX Series Router.
• Links between MX Series Routers and all SRX Series Firewalls are configured with two eBGP sessions. One for TRUST and one for UNTRUST.
• The load balancing policy with source-hash for route 0/0 is configured in the forwarding table.
• The load balancing policy with destination-hash for client prefix routes (users) is configured in the forwarding table.
• The default 0/0 route is received by all the SRX Series Firewalls on UNTRUST side and advertised using eBGP to MX Series Router on the TRUST side. The MX Series Router imports this route on the TRUST instance using load balancing CHASH policy.
• Client prefix route is received by all the SRX Series Firewalls on TRUST side and advertised using eBGP to MX Series Router on the UNTRUST side. The MX Series Router imports this route on the UNTRUST instance using load balancing CHASH policy.
• The MX Series Router on the TRUST side has all the ECMP routes for 0/0 route.
• The MX Series Router on the UNTRUST side has all the ECMP routes for the client prefix routes.
• Forward traffic flow from client to server reaches MX Series Router on TRUST instance and hits 0/0 route and takes any one ECMP next-hop to SRX series firewall based on the calculated source IP based hash value.
• The SRX Series Firewalls creates an SFW flow session and routes the packet to MX Series Router on the UNTRUST direction towards the server.
• Reverse traffic flow from server to client reaches MX Series Router on UNTRUST instance and hits client prefix route and takes the same ECMP next hop based on the calculated destination IP based hash value.
• Since the five tuples of the SFW sessions do not change, calculated hash value remains the same and takes the same ECMP next hop/SRX Series Firewalls on the forward and reverse flow. This makes sure symmetricity is maintained in the SRX Series Firewalls.
• When any SRX Series Firewall goes down, CHASH on the MX Series Router ensures that the sessions on the other SRX Series Firewalls are not disturbed and only sessions on the down SRX Series Firewalls are redistributed.

ECMP/CHASH in Topology 1 (Single MX, scale-out SRXs) for CGNAT:

Figure 8: Topology 1 - ECMP CHASH - CGNAT Use Case

• The SRX Series Firewalls are deployed in a standalone scaled out devices to a single MX Series Router.
• Links between the MX Series Router and SRX Series Firewalls are configured with two eBGP sessions. One for TRUST and one for UNTRUST.
• Unique NAT pool IP address ranges are allocated per SRX Series Firewalls.
• The load balancing policy with source-hash for route 0/0 is configured in the forwarding table.
• 0/0 route is received by the SRX Series Firewalls on the Untrust side and is advertised using eBGP to MX Series Router on the TRUST side. The MX Series Router imports this route on the TRUST instance using load balancing CHASH policy.
• Client prefix route is received by the SRX Series Firewalls on the TRUST side and NAT pool route prefix is advertised using eBGP to MX Series Router on the UNTRUST side.
• The MX Series Router on the TRUST side has an ECMP route for 0/0 route.
• The MX Series Router on the UNTRUST side has a unique route for the NAT pool route prefix.
• The forward traffic flow from client to server reaches the MX Series Router on TRUST instance and hits 0/0 route. It takes any one ECMP next-hop to SRX Series Firewalls based on the calculated source IP based hash value.
• The SRX Series Firewalls creates an NAT flow session and routes the packet to MX Series Router on the UNTRUST direction towards the server.
• Reverse traffic flow from server to client reaches MX Series Router on UNTRUST instance and hits unique NAT pool prefix route and takes the same SRX Series Firewalls where forward flow is anchored. This makes sure symmetricity is maintained in the SRX devices.
• When any SRX Series Firewall goes down, CHASH on the MX Series Router ensures that the sessions on the other SRX Series Firewalls are not disturbed and only sessions on the down SRX Series Firewalls are redistributed.

ECMP/CHASH in Topology 2 (Dual MX, SRX MNHA Pairs) for SFW:

Figure 9: Topology 2 - ECMP CHASH - SFW Use Case

• When the SRX Series Firewalls are deployed in pair with MNHA, session syncs both ways depending on where the traffic is received.
• The MX Series Router pair is configured with SRD redundancy for user management of the MX HA pair.
• The MX Series Router pair monitor links towards Trust GW / Internet GW router and links between the MX Series Router to the SRX Series Firewalls. SRD triggers automatic switch over to another MX Series Router if any of this link fails. It can also failover when MX304-1 completely goes down. The MX Series Routers have 4x100G interface connected to the SRX4600 devices as an AE bundle and contain three VLANs (trust, untrust and HA management).MX304-1 remains primary ECMP path and MX304-2 standby ECMP path.
• SRD is used for MX Series Router redundancy and controls the MX master ship state transition. It also installs a signal route on the master MX Series Router which is used for route advertisement with preference.
• MX304-1 advertises routes as it is, whereas MX304-2 standby advertises routes with as-path-prepend.
• Interfaces on MX304-1 towards SRX Series Firewalls and MX304-2 towards SRX Series Firewalls need to be provisioned using similar interface numbering with similar I/O card. This helps in maintaining the same unilist next-hop ordering on both the MX304-1 and MX304-2 routers. RPD decides unilist next-hop ordering based on the interface ifl index number (Ascending order of interface ifl numbers).
• Since unilist next-hop ordering is same in both MX Series Router, post any MX Series Router switchover, there is not going to be any issue with hash (source or destination).
• If any failure is detected by an active MX Series Router (SRD), the failover to the other MX Series Router. This implies that all traffic reaches this second MX Series Router (Second MX Series Router has taken ownership of the SRX Series Firewalls and announced around the routes to itself). It also implies that traffic to SRX Series Firewalls connected to MX304-1 is sent to SRXs connected to MX304-2. This is a complete failover of the top architecture to the bottom one.

The following MX Series Router configuration shows how the SRD process monitors events to decide any release or acquisition of mastership. On the SRD process side, the relevant configuration contains:

On the routing side, the SRD configuration looks for the existence of specific route and then announces the default route conditionally:

Traffic Load Balancer Overview

This feature relates to topology 3 (single MX Series Router, scale-out SRX MNHA pairs) and topology 4 (dual MX Series Routers and scale-out SRX MNHA pairs).

Figure 10: Topology 3 and 4 - TLB - SFW Use Case

Traffic Load Balancer in MX Router:

Traffic Load Balancer (TLB) functionality provides stateless translated or non-translated traffic load balancer, as an inline PFE service in the MX Series Routers. Load balancing in this context is a method where incoming transit traffic is distributed across configured servers that are in service. This is a stateless load balancer, as there is no state created for any connection, and so there are no scaling limitations. Throughput could be close to line rate. TLB has two modes of load balancing i.e., translated (L3) and non-translated Direct Server Return (L3).

For the scale-out solution, the TLB mode non-translated Direct Server Return (L3) is used. As part of TLB configuration, there is a list of available SRX Series Firewalls addresses and the MX PFE programs a selector table based on this SRX Series Firewalls. TLB does a health check (ICMP usually however it can do HTTP, UDP, and TCP checks) for each of the SRX Series Firewalls individually. TLB health check is done using MX Series Router routing engine. If the SRX Series Firewalls pass the health check, TLB installs a specific IP address route or wild card IP address (TLB config option) route in the routing table with next-hop as composite next-hop. Composite next-hop in the PFE is programmed with all the available SRX Series Firewalls in the selector table. Filter based forwarding is used to push the "Client to Server" traffic to the TLB where it hits the TLB installed specific IP address route or wild card IP address route to get the traffic sprayed across the available SRX Series Firewalls with source or destination hash. "Server to Client" is directly routed back to client instead of going through the TLB.

Figure 11: TLB Work in RE and PFE

Using TLB in the MX Router for the Scale-Out SRX Solution with SFW:

Figure 12: Topology 3 - Scale-Out SFW with TLB

• All the SRX Series Firewalls are configured with BGP to establish an eBGP peering sessions with MX Series Router nodes.
• The MX Series Router is configured with TLB on the Trust routing instance to do the load balancing of data traffic coming from client-side gateway router towards scaled out SRX Series Firewalls.
• All the scale-out SRX Series Firewalls connected to MX Series Router are configured with unique IP address (for example, loopback) which is used by MX TLB to do the health check and build up the selector table in the PFE. PFE uses this selector table to load balance the packet across the available next hops. This health check is reachable through BGP connection.
• Filter based forwarding based on source IP address match is used in MX Series Router to push SFW specific traffic to the TLB trust forwarding instance.
• TLB forwarding instance has a default route with next-hop as list of SRX Series Firewalls. TLB installs this default route when its health check passes with at least one SRX Series Firewalls.
• TLB does source based hash load balancing across all the available SRX next-hop devices.
• Load balanced SFW data sessions are anchored on any available SRX Series Firewalls and SFW flow gets created. Then it is routed to reach the server through MX Series Router over Untrust routing instance.
• For the return traffic coming from server to client direction on the MX Untrust routing instance, another TLB instance is configured on MX Untrust routing instance to do the load balancing back to the same SRX Series Firewalls.
• Filter based forwarding of destination IP address match is used in MX Series Router to push SFW specific traffic to the TLB UNTRUST forwarding instance.
• TLB forwarding instance has a default route with next-hop as list of SRX Series Firewalls. TLB installs this default route when its health check passes with at least one SRX Series Firewalls.
• TLB does destination-based hash load balancing across all the available SRX next-hop devices.
• Load balanced SFW data sessions are load balanced to the same SRX Series Firewalls on the return direction and uses the same flow to reach the client through MX Series Router over TRUST routing instance.

Using TLB in the MX Router for the Scale-Out SRX Solution with CGNAT:

Figure 13: Topology 3 - Scale-Out CGNAT with TLB

• All the scale-out SRX Series Firewalls connected to MX Series Routers are configured with BGP connections.
• Each scaled out SRX Series Firewall needs to have a unique NAT pool range, and this must be advertised towards the MX Untrust direction. (This is the main difference with SFW use case, as it needs to announce the NAT pools)
• The MX Series Router is configured with TLB on the Trust routing instance to do the load balancing of data traffic coming from client-side gateway router towards scaled out SRX Series Firewalls.
• All the scale-out SRX Series Firewalls connected to the MX Series Router are configured with unique IP address, which is used by MX TLB to do the health check and build the selector table in the PFE. PFE uses this selector table to load balance the packet across the available next hops. This health check is reachable through BGP connection.
• The filter-based forwarding on source IP address match is used in the MX Series Router to push the NAT specific traffic to the TLB trust forwarding instance.
• The TLB forwarding instance has a default route with next-hop as list of SRX Series Firewalls. TLB installs this default route when its health check passes with at least one SRX Series Firewalls.
• TLB does source-based hash load balancing across all the available SRX next-hop devices.
• Load balanced NAT data sessions are anchored on any available SRX Series Firewalls and NAT flow gets created. Then it is routed to reach the server through MX Series Router over UNTRUST routing instance.
• For the return traffic coming from server to client direction on the MX Untrust routing instance, Unique NAT pool routes are used to route the traffic to the same SRX devices.
• The SRX Series Firewalls use same NAT flow to process the return traffic and route the packet towards MX Series Router on the TRUST direction. The MX Series Router routes the packet back to the client.

Configuration Examples for ECMP CHASH

The following sample configurations are proposed to understand the elements making this solution work, including configurations for both MX Series Router and some SRX Series Firewalls. It contains a lot of repetitive statements. It shows Junos hierarchical view.

Source-hash for forward flow and destination-hash for reverse flow is common for all ECMP based solutions or TLB based solutions. Consistent hash (CHASH) is used during any next-hop failure where it helps an existing session on an active next-hop to remain undisturbed, while sessions on down next-hop is redistributed over other active next-hop. This CHASH behavior is pre-built in the TLB solution. However, in ECMP based solution you must configure this CHASH configuration explicitly using BGP import policy.

The following sample MX Series Router configuration is for ECMP load balancing using source and destination hash:

The following MX Series Router configuration is an example for specific forward and return traffic with CHASH:

The following MX Series Router configuration is for the routing instance and BGP peering with default GW and the SRX Series Firewall:

The following is the sample SRX1 configuration for SFW and CGNAT:

While testing these use cases, some outputs for ECMP CHASH shows the following route selections:

Configuration Example for TLB

Like ECMP CHASH, the trust-vr/untrust-vr are similar in the TLB use case, with BGP peering SRX Series Firewalls on each side, however different configuration needs to be made for the TLB services, including additional routing-instances and less policy statements.

Source-hash for forward flow and destination-hash for reverse flow is common for all ECMP based or TLB based solutions. Consistent hash is used during any next-hop failures where it helps an existing session on active next-hops to remain undisturbed, while sessions on down next-hops get redistributed over other active next-hops. This CHASH behavior is pre-built in the TLB solution.

Following sample configuration shows general load balancing strategy for anything but TLB:

The following sample configuration of MX Series Router is for specific forward and return traffic:

The following sample configuration shows that how traffic is redirected to TLB instance using filter-based forwarding (associated with routing-instance srx_mnha_group_tlb-fi):

The following sample configuration shows interface loopbacks used by TLB for health checking to the SRXs:

And the following sample configuration is of the TLB service part (for example, with a NAT service, only trust side TLB instance is used as NAT Pools are announced for return traffic):

The following sample configuration is of SRX1 for SFW:

While running tests, some output for TLB could be seen as the group usage and packets/bytes to each SRX Series Firewalls:

Common Configurations for ECMP CHASH and TLB

Some elements of configuration need to be in place for both load balancing methods. The following sample configurations are for TRUST and UNTRUST VR and the peering with each SRX Series Firewalls. It also shows some other less seen configuration elements.

• Following is some of the common configurations when using dual MX Series Router topology:Both MX Series Router calculate same hash value when both have same number of next hops, however this is added in Junos OS Release 24.2 (hidden before).