ON THIS PAGE
Introduction to SRX Series Firewall Multi-Node High Availability
Equal-cost Multipath /Consistent Hashing Load Balancing Overview
Equal-cost Multipath /Consistent Hashing (CHASH) in MX Series Router:
ECMP/CHASH in Topology 1 (Single MX Series Router, Scale-Out SRXs) for SFW:
ECMP/CHASH in Topology 1 (Single MX Series Router, scale-out SRXs) for Source NAT:
ECMP/CHASH in Topology 2 (Dual MX Series Router, SRX MNHA Pairs) for SFW:
Using TLB in the MX Series Router for the Scale-Out SRX Solution with SFW:
Using TLB in the MX Series Router for the scale-out SRX Solution with Source NAT:
Solution Details
Traffic Path in SFW Scale-Out Solution
The Scale-Out solution is based on BGP as dynamic routing protocol. It enables all the MX routers and SRX Series Firewalls to learn their surrounding networks. However, most importantly to exchange path information for the network traffic that needs to be sent from MX Series Router across each SRX Series Firewall to the next MX Series Router. This protocol enables exchange of network paths for the internal/user subnets and the default/specific external network. When each SRX Series Firewall announces what it has learned from the other side, each with the same network cost, the load balancing can then use those routes for load balancing traffic across each SRX Series Firewall.
Figure 2 shows how traffic flows from an MX Series Router to multiple SRX Series Firewalls using ECMP load balancing method. You can see that SRX Series Firewalls are in a symmetric sandwich between the two MX Series Routers, whether those MX Series Router are a single physical node configured with two routing instances (more typical) or two physical MX Series Router nodes on each side. The routing principle remains the same as if two routing nodes are used, maintaining traffic flow distribution that is consistent in both directions.
The MX Series Router on the left has a TRUST VR routing-instance used to forward traffic to each SRX Series Firewalls.
The MX Series Router on the right has the symmetric UNTRUST VR to receive traffic from each SRX Series Firewalls and forward it to the next hop toward the target resources. The routes on each side are announced through BGP to the next hop, making its path available on each MX Series Router instance through each SRX Series Firewalls (with same cost for load balancing to happen).
Routes are announced through BGP, each MX Series Router with their own BGP Autonomous System (AS) and peer with the SRX Series Firewalls on their two sides (TRUST and UNTRUST zones in a single routing instance). MX Series Router may peer with any other routers bringing connectivity to the clients and servers (here GW Router).
When the routes across each SRX Series Firewalls is known to have similar cost, then the Load Balancing method is used as explained below.
For SNAT use case, this is very similar to SFW. However, the NAT pools are exchanged on the right MX Series Router for the return traffics to flow back to the correct SRX Series Firewalls:
Introduction to SRX Series Firewall Multi-Node High Availability
To begin with, refer to this extract from the public documentation on MNHA, https://www.juniper.net/documentation/us/en/software/junos/high-availability/topics/topic-map/mnha-introduction.html
Juniper Networks® SRX Series Firewalls support a new solution, Multi-node High Availability (MNHA), to address high availability requirements for modern data centres. In this solution, both the control plane and the data plane of the participating devices (nodes) are active at the same time. Thus, the solution provides inter chassis resiliency.
The participating devices are either co-located or physically separated across geographical areas or other locations such as different rooms or buildings. Having nodes with high availability across geographical locations ensures resilient service. If a disaster affects one physical location, MNHA fails over to a node in another physical location, thereby ensuring continuity.
In MNHA, both SRX Series Firewalls have an active Control Plane and communicate their status over an inter chassis link (ICL) that can be either direct or routed across the network, allowing both to be separated in distance and keep synchronizing sessions and IKE security associations. Also, they do not share a common configuration, and this enable different IP settings on both SRX Series Firewalls. However, some commit sync mechanism that can be used for the elements of configuration that need to be same on both the platforms.
The SRX Series Firewalls use a SRG for the Data Plane that can be either Active or Backup (for SRG1 and above). An exception is the SRG group 0 (zero) that is always active on both SRX Series Firewalls. This group can be used natively by scale-out to load balance traffic across both SRX Series Firewalls at the same time. However, some interest exists for the other modes where it can be active or backup for SRG1 and backup or active for SRG2, which is like active SRG0. However, SRG2 can add some routing information (like BGP as-path-prepend) under certain conditions. SRG1/+ offers more health checking of its surrounding environment that can be leveraged to make a SRGn group active, backup, or ineligible.
MNHA can select any mode from the following network modes:
- Default Gateway or L2 mode: It uses same network segment at L2 on different sides of the SRX Series Firewalls (e.g. trust/untrust) and both SRX Series Firewalls share a common IP or MAC addresses on each network segment. It does not mean the SRX Series Firewallsis in switching mode, it does route between its interfaces. However, but shares the same broadcast domain on one side with the other SRX Series Firewalls, and same on the other side.
- Hybrid mode or mix of L2 and L3: This mode uses L2 and IP address on one side of the SRX Series Firewalls (e.g. trust) and routing on the other side of the SRX Series Firewalls (e.g. untrust). On L3 side, both firewalls are on different subnets, and they share same subnet on the L2 side with a common VIP like 10.0.0.1/24.
- Routing mode or L3: This architecture is used for this JVD where each side of the SRX Series Firewalls is using a different IP address. Even between the SRX Series Firewalls, there is no common IP subnet. and all communication with the rest of the network is done through routing. This mode is perfect for scale-out communication using BGP with the MX Series Router.
Whether to use SRG0 active-active, or SRG1 active-backup (single one active at a time), or a combination of SRG1 active-backup and SRG2 backup
Equal-cost Multipath /Consistent Hashing Load Balancing Overview
This feature relates to topology 1 (single MX Series Router, scale-out SRXs) and topology 2 (dual Active/Passive MX and scale-out MNHA SRX Series Firewall pairs).
Equal-cost Multipath /Consistent Hashing (CHASH) in MX Series Router:
Equal-cost multipath (ECMP) is a network routing strategy that allows traffic of the same session, or traffic with the same source and destination — to transmit across multiple paths of equal cost. It is a mechanism that provides load balancing of traffic and increases bandwidth by fully utilizing the unused bandwidth on links to the same destination.
When forwarding a packet, the routing technology decides the next hop path to use. In deciding the path, the device takes into account the packet header fields that identify a flow. When ECMP is used, next hop paths of equal cost are identified based on routing metric calculations and hash algorithms. That is, routes of equal cost have the same preference and metric values, and the same cost to the network. The ECMP process identifies a set of routers, each of which is a legitimate cost of next hop towards the destination. The identified routes are called an ECMP set. An ECMP set is formed when the routing table contains multiple next hop addresses for the same destination with equal cost (routes of equal cost have the same preference and metric values). If there is an ECMP set for the active route, Junos OS uses a hash algorithm to choose one of the next hop addresses in the ECMP set to install in the forwarding table. You can configure Junos OS so that multiple next hop entries in an ECMP set are installed in the forwarding table. On Juniper Networks devices, per-packet load balancing can be performed to spread traffic across multiple paths between routing devices.
Example of learned routes and forwarding table for the same destination (assuming traffic target is within 100.64.1.0/24 and SRX Series Firewalls BGP peers are 10.1.1.0, 10.1.1.8, and 10.1.1.16):
jcluser@mx-01> show route 100.64.1.0/24
trust-vr.inet.0: 30 destinations, 33 routes (30 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
100.64.1.0/24 *[BGP/170] 4d 04:52:53, MED 10, localpref 100
AS path: 000 64500 I, validation-state: unverified
to 10.1.1.0 via ae1.0 ## learning routes from peer SRX1
> to 10.1.1.8 via ae2.0 ## learning routes from peer SRX2
to 10.1.1.16 via ae3.0 ## learning routes from peer SRX3
jcluser@mx-0> show route forwarding-table destination 100.64.1.0/24 table trust-vr
Routing table: trust-vr.inet
Internet:
Destination Type RtRef Next hop Type Index NhRef Netif
100.64.1.0/24 user 0 ulst 1048580 2
10.1.1.0 ucst 801 4 ae1.0 ## path to SRX1
10.1.1.8 ucst 798 5 ae2.0 ## path to SRX2
10.1.1.16 ucst 799 5 ae3.0 ## path to SRX3With scale-out architecture where stateful security devices are connected, maintaining symmetricity of the flows in the security devices is the primary objective. The symmetricity means traffic from a subscriber (user) and to the subscriber must always reach the same server (which maintains the subscriber state). To reach the same server, the traffic must be hashed onto the same link towards that server for traffic in both directions.
A subscriber is identified by the source IP address in the upstream direction (client to server) and by the destination IP address in the downstream direction (server to client). The MX Series Routers do symmetric hashing i.e. for a given (sip, dip) tuple, same hash is calculated irrespective of the direction of the flow i.e. even if sip and dip are swapped. However, the requirement is that all flows from a subscriber reach the same SRX Series Firewall, so you need to hash only source IP address (and not destination IP address) in one direction and vice versa in the reverse direction.
By default, when a failure occurs in one or more paths, the hashing algorithm recalculates the next hop for all paths, typically resulting in the redistribution of all flows. Consistent load balancing enables you to override this behavior so that only flows for inactive links are redirected. All existing active flows are maintained without disruption. In such an environment, the redistribution of all flows when a link fails potentially results in significant traffic loss or a loss of service to an SRX Series Firewall whose links remain active. However, consistent load balancing maintains all active links and remaps only those flows affected by one or more link failures. This feature ensures that flows connected to links that remain active continue uninterrupted.
This feature applies to topologies where members of an ECMP group are external BGP neighbors in a single-hop BGP session. Consistent load balancing does not apply when you add a new ECMP path or modify an existing path in any way. New SRX add design is implemented recently where you can add SRX Series Firewalls gracefully with an intent of equal redistribution from each active SRX series firewall, hence causing minimal impact to the existing ECMP flows. For example, if there are four active SRX Series Firewalls carrying 25% of total flows on each link and the fifth SRX Series Firewall (previously unseen) is added, 5% of flows from each existing SRX Series Firewall moves to the new SRX series firewall. Hence making 20% of flow redistribution from existing 4 SRX Series Firewalls to new one.
The following information shares details for each step of route exchange between MX Series Router and SRXs, traffic flows, for each use case.
ECMP/CHASH in Topology 1 (Single MX Series Router, Scale-Out SRXs) for SFW:
- SRX Series Firewalls are deployed in a standalone scaled out devices to single MX Series Router.
- Links between MX Series Routers and all SRX Series Firewalls are configured with two eBGP sessions. One for TRUST and one for UNTRUST.
- The Load balancing policy with source-hash for route 0/0 is configured in the forwarding table.
- The Load balancing policy with destination-hash for client prefix routes (users) is configured in the forwarding table.
- The default 0/0 route is received by all the SRX Series Firewalls on UNTRUST side and advertised using eBGP to MX Series Router on the TRUST side. The MX Series Router imports this route on the TRUST instance using load balancing CHASH policy.
- Client prefix route is received by all the SRX Series Firewalls on TRUST side and advertised using eBGP to MX Series Router on the UNTRUST side. The MX Series Router imports this route on the UNTRUST instance using load balancing CHASH policy.
- The MX Series Router on the TRUST side has all the ECMP routes for 0/0 route.
- The MX Series Router on the UNTRUST side has all the ECMP routes for the client prefix routes.
- Forward traffic flow from client to server reaches MX Series Router on TRUST instance and hits 0/0 route and takes any one ECMP next-hop to SRX Series Firewall based on the calculated source IP based hash value.
- The SRX Series Firewalls creates an SFW flow session and routes the packet to MX Series Router on the UNTRUST direction towards the server.
- Reverse traffic flow from server to client reaches MX Series Router on UNTRUST instance and hits client prefix route and takes the same ECMP next hop based on the calculated destination IP based hash value.
- Since the five tuples of the SFW sessions do not change, calculated hash value remains the same and takes the same ECMP next hop/SRX Series Firewalls on the forward and reverse flow. This makes sure symmetricity is maintained in the SRX Series Firewalls.
- When any SRX Series Firewall goes down, CHASH on the MX Series Router ensures that the sessions on the other SRX Series Firewalls are not disturbed and only sessions on the down SRX Series Firewalls are redistributed.
ECMP/CHASH in Topology 1 (Single MX Series Router, scale-out SRXs) for Source NAT:
- The SRX Series Firewalls are deployed in a standalone scaled out devices to a single MX Series Router.
- Links between the MX Series Router and SRX Series Firewalls are configured with two eBGP sessions. One for TRUST and one for UNTRUST.
- Unique NAT pool IP address ranges are allocated per SRX Series Firewalls.
- The load balancing policy with source-hash for route 0/0 is configured in the forwarding table.
- 0/0 route is received by the SRX Series Firewalls on the Untrust side and is advertised using eBGP to MX Series Router on the TRUST side. The MX Series Router imports this route on the TRUST instance using load balancing CHASH policy.
- Client prefix route is received by the SRX Series Firewalls on the TRUST side and NAT pool route prefix is advertised using eBGP to MX Series Router on the UNTRUST side.
- The MX Series Router on the TRUST side has an ECMP route for 0/0 route.
- The MX Series Router on the UNTRUST side has a unique route for the NAT pool route prefix.
- The forward traffic flow from client to server reaches the MX Series Router on TRUST instance and hits 0/0 route. It takes any one ECMP next-hop to SRX Series Firewalls based on the calculated source IP based hash value.
- The SRX Series Firewalls creates an NAT flow session and routes the packet to MX Series Router on the UNTRUST direction towards the server.
- Reverse traffic flow from server to client reaches MX Series Router on UNTRUST instance and hits unique NAT pool prefix route and takes the same SRX Series Firewalls where forward flow is anchored. This makes sure symmetricity is maintained in the SRX Series Firewalls.
- When any SRX Series Firewall goes down, CHASH on the MX Series Router ensures that the sessions on the other SRX Series Firewalls are not disturbed and only sessions on the down SRX Series Firewalls are redistributed.
ECMP/CHASH in Topology 2 (Dual MX Series Router, SRX MNHA Pairs) for SFW:
- A session where SRX Series Firewalls are deployed in pair with MNHA syncs both ways depending on where the traffic is received.
- The MX Series Router pair is configured with SRD redundancy for user management of the MX Series Routers pair.
- The MX Series Router pair monitors the links towards TRUST or INTERNET GW router and links between MX Series Router to SRX Series Firewall. If any of these links fails, SRD triggers automatic switch over to other MX Series Routers. It can also failover when MX304-1 completely goes down.
- The MX Series Routers have 4x100G interface connected to the SRX4600 devices as an AE bundle and contain three VLANs (trust, untrust, and HA management).
- MX304-1 remains a primary ECMP path and MX304-2 remains standby ECMP path.
-
SRD is used for MX Series Router redundancy and controls the MX Series Router master ship state transition. It also installs a signal route on the master MX Series Router which is used for route advertisement with preference.
MX304-1 master advertises routes as it is, whereas MX304-2 standby advertises routes with as-path-prepend.
- Interfaces on MX304-1 towards SRX Series Firewall and MX304-2 towards SRX Series Firewall need to be provisioned using similar interface numbering with similar I/O card. This helps in maintaining the same unilist next-hop ordering on both the MX304-1 and MX304-2 routers. RPD decides unilist next-hop ordering based on the interface ifl index number (Ascending order of interface ifl numbers).
- As unilist next-hop ordering is same in both the MX Series Routers, it does not cause any issue with hash (source or destination) post any MX Series Router switchover.
- In case a failure is detected by an active MX Series Router (SRD), the failover to the other MX device implies that all traffic may reach this second MX Series Router (it takes the ownership of the SRX Series Firewalls and announces routes to itself). It also implies that traffic to the SRX Series Firewalls connected to MX304-1 is sent to SRX Series Firewalls connected to MX304-2. This is a complete failover from the top architecture to the bottom one.
The following MX Series Router configuration shows how the SRD process monitors events to decide any release or acquisition of master ship. On the SRD process side, the relevant configuration contains:
event-options {
redundancy-event 1_MSHIP_ACQUIRE_EVENT {
monitor {
peer { ### Monitored membership released from MX peer
mastership-release;
}
}
}
redundancy-event 1_MSHIP_RELEASE_EVENT {
monitor {
link-down { ### Monitored interfaces when link is down
ae1;
ae1.0;
ae1.1;
...
...
ae11.80;
}
process { ### Monitored process restarting
routing {
restart;
}
}
peer { ### Monitored membership acquisition from MX peer
mastership-acquire;
}
}
}
}
services {
redundancy-set {
1 {
redundancy-group 1;
redundancy-policy [ 1_ACQU_MSHIP_POL 1_RELS_MSHIP_POL ];
}
}
}
policy-options {
redundancy-policy 1_ACQU_MSHIP_POL {
redundancy-events 1_MSHIP_ACQUIRE_EVENT;
then {
acquire-mastership;
add-static-route 10.3.0.3/32 {### Adds this route when acquiring mastership
receive;
routing-instance SRD;
}
}
}
redundancy-policy 1_RELS_MSHIP_POL {
redundancy-events 1_MSHIP_RELEASE_EVENT;
then {
release-mastership;
delete-static-route 10.3.0.3/32 {### Removes this route when releasing mastership
routing-instance SRD;
}
}
}
}On the routing side, the SRD configuration looks for the existence of specific route and then announces the default route conditionally:
interfaces {
ae10 {
description To-GW;
vlan-tagging;
unit 40 {
description "GI-POC MX-GW TRUST_VR_v4";
vlan-id 40;
family inet {
address 10.40.1.2/24 {
vrrp-group 40 {
virtual-address 10.40.1.1;
...
...
track {
route 10.3.0.3/32 routing-instance SRD priority-cost 30;
}
}
}
}
}
unit 80 {
description "GI-POC MX-GW UNTRUST_VR_v4";
vlan-id 80;
family inet {
address 10.80.1.2/24 {
vrrp-group 80 {
virtual-address 10.80.1.1;
...
...
track {
route 10.3.0.3/32 routing-instance SRD priority-cost 30;
}
}
}
}
}
}
}
policy-options {
condition 1_ROUTE_EXISTS {
if-route-exists { ### If this route exists….
10.3.0.3/32;
table SRD.inet.0;
}
}
policy-statement MX-to-MX-1_trust_export {
term 1 {
from {
protocol bgp;
route-filter 0.0.0.0/0 exact; ### Then announces default route to self on trust-vr
condition 1_ROUTE_EXISTS;
}
then {
local-preference 200;
next-hop self;
accept;
}
}
term 2 {
then reject;
}
}
policy-statement MX-to-MX-1_untrust_export {
term 1 {
from {
protocol bgp; ### Then announces clients route to self on untrust-vr
prefix-list-filter GI-FW_clients_v4 exact;
condition 1_ROUTE_EXISTS;
}
then {
local-preference 200;
next-hop self;
accept;
}
}
term 2 {
then reject;
}
}
}
routing-instances {
1_TRUST_VR {
instance-type virtual-router;
protocols {
bgp {
group MX-TO-MX-IBGP {
type internal;
export MX-to-MX-1_trust_export; ### Export conditional route to external GW
...
}
}
}
...
...
interface ae10.40;
}
1_UNTRUST_VR {
instance-type virtual-router;
protocols {
bgp {
group MX-TO-MX-IBGP {
type internal;
export MX-to-MX-1_untrust_export; ### Export conditional route to external GW
...
}
}
}
...
...
interface ae10.80;
}
SRD {
instance-type virtual-router;
...
}
} Traffic Load Balancer Overview
This feature relates to topology 3 (single MX Series Router, scale-out SRX MNHA pairs) and topology 4 (dual MX Series Router and scale-out SRX MNHA pairs).
Traffic Load Balancer Service in MX Series Router:
The Traffic Load Balancer (TLB) functionality provides a stateless translated or non-translated traffic load balancer, as an inline PFE service in the MX Series Routers. Load balancing in this context is a method where incoming transit traffic is distributed across configured servers that are in service. This is a stateless load balancer, as no state is created for any connection, and there are no scaling limitations. Throughput could be close to line rate. TLB has two modes of load balancing i.e., translated (L3) and non-translated Direct Server Return (DSRL3).
For the scale-out solution, the TLB mode non-translated Direct Server Return (L3) is used. As part of TLB configuration,there is a list of available SRX Series Firewalls addresses and the MX Series Router PFE programs a selector table based on this SRX Series Firewalls. TLB does a health check (ICMP usually however, it can do HTTP, UDP, and TCP checks) for each of the SRX Series Firewalls individually. TLB health check is done using MX Series Router routing engine. If health check pass for any of the SRX Series Firewalls, TLB installs a specific IP route or wild card IP address (TLB config option) route in the routing table with next-hop as composite next-hop. Composite next-hop in the PFE is programmed with all the available SRX Series Firewalls in the selector table. Filter based forwarding is used to push the "Client to Server" traffic to the TLB where it hits the TLB installed specific IP address route or wild card IP address route to get the traffic sprayed across the available SRX Series Firewalls with source or destination hash. "Server to Client" is directly routed back to the client instead of going through the TLB.
TLB has existed in the Junos and MX Series Router family for few years now (as early as Junos OS Release 16.1R6) and is used successfully on large server farms with around 20,000 servers. TLB is using the control part and the health check on MS-MPC service cards on MX240/480/960/MX2000 chassis before, data plane/PFE is already on the line cards. It is not running on the RE as it has been now implemented on MX304/MX10000 chassis. For more information see, https://www.juniper.net/documentation/us/en/software/junos/interfaces-next-gen-services/interfaces-adaptive-services/topics/concept/tdf-tlb-overview.html.
Using TLB in the MX Series Router for the Scale-Out SRX Solution with SFW:
- All the SRX Series Firewalls are configured with BGP to establish an eBGP peering sessions with MX Series Routers.
- The MX Series Routers are configured with TLB on the TRUST routing instance to do the load balancing of data traffic coming from the client-side gateway router towards scaled out SRX Series Firewalls.
- All the scale-out SRX Series Firewalls connected to the MX Series Router are configured with unique IP addresses (for example, loopback) which are used by MX Series Router TLB to do the health check and build up the selector table in the PFE. PFE uses this selector table to load balance the packet across the available next hops. This health check is reachable through BGP connection.
- Filter based forwarding based on source IP address match is used in the MX Series Router to push SFW specific traffic to the TLB trust forwarding instance.
- TLB forwarding instance has a default route with next-hop as list of SRX Series Firewalls. TLB installs this default route when its health check passes with at least one SRX Series Firewall.
- TLB does source-based-hash load balancing across all the available SRX next hop Series Firewalls.
- Load balanced SFW data sessions are anchored on any available SRX Series Firewalls and create SFW flow. Then it’s routed to reach the server through MX Series Router over UNTRUST routing instance.
- For the return traffic coming from server to client direction on the MX Series Router UNTRUST routing instance, another TLB instance is configured on MX Series Router UNTRUST routing instance to do the load balancing back to the same SRX Series Firewalls.
- Filter based forwarding of destination IP address match is used in the MX Series Router to push SFW specific traffic to the TLB UNTRUST forwarding instance.
- TLB forwarding instance may have a default route with next-hop as list of SRX Series Firewalls. TLB installs this default route when its health check passes with at least one SRX Series Firewall.
- TLB does destination-based-hash load balancing across all the available SRX next-hop Series Firewalls.
- Load balanced SFW data sessions are load-balanced to the same SRX Series Firewalls on the return direction and uses the same flow to reach the client through MX Series Router over TRUST routing instance.
Using TLB in the MX Series Router for the scale-out SRX Solution with Source NAT:
- All the scale-out SRX Series Firewalls connected to the MX Series Router are configured with BGP connections.
- Each scaled out SRX Series Firewall needs to have a unique NAT pool range, and this must be advertised towards the MX Series Router UNTRUST direction. (This is the main difference with SFW use case, as it needs to announce the NAT pools).
- The MX Series Router is configured with TLB on the TRUST routing instance to do the load balancing of data traffic coming from the client-side gateway router towards scaled out SRX Series Firewalls.
- All the scale-out SRX series firewalls connected to the MX Series Router are configured with unique IP address which is used by MX Series Router TLB to do the health check and build up the selector table in the PFE. PFE uses this selector table to load balance the packet across the available next hops. This health check is reachable through BGP connection.
- Filter based forwarding where source IP address match is used in the MX Series Router to push the NAT specific traffic to the TLB trust forwarding instance.
- TLB forwarding instance has a default route with next-hop as list of SRX Series Firewalls. TLB installs this default route when its health check passes with at least one SRX Series Firewall.
- TLB does source-based-hash load balancing across all the available SRX next hop Series Firewall.
- Load balanced NAT data sessions are anchored on any available SRX Series Firewall and create an NAT flow. Then it’s routed to reach the server through MX Series Router over UNTRUST routing instance.
- For the return traffic coming from server to client direction on the MX Series Router UNTRUST routing instance, a unique NAT pool route is used to route the traffic to the same SRX Series Firewall.
- The SRX device uses the same NAT flow to process the return traffic and routes the packet towards MX Series Router in the TRUST direction. The MX Series Router routes the packet back to the client.
Configuration Examples for ECMP CHASH
The following sample configurations can be used to understand the elements making this solution work, including configurations for both the MX Series Router and some SRX Series Firewalls. It contains a lot of repetitive statements. It shows Junos hierarchical view.
These configuration examples can be used for IPv6.
Also, the TRUST side (or left/client) is colored in green to make it clearer, and the UNTRUST side (or right/Internet) is in blue. A few other colors are used to highlight some other important elements, however, are not related to sides of the solution.
Source-hash for forward flow and destination-hash for reverse flow is common for all ECMP based or TLB based solutions. Consistent hash (CHASH) is used during any next-hop failures where it helps an existing session on an active next-hop to remain undisturbed, while sessions on down next-hop is redistributed over other active next-hop. This consistent hash behavior is pre-built in the TLB solution. However, in ECMP based solution you must configure this consistent hashing configuration explicitly using BGP import policy.
The following sample MX Series Router configuration is for ECMP load balancing using source and destination hash:
policy-options {
prefix-list clients_v4 { ### clients subnet(s)
192.0.2.0/25;
}
policy-statement pfe_lb_hash {
term source_hash {
from {
route-filter 0.0.0.0/0 exact;
}
then {
load-balance source-ip-only; ### when 0/0, then LB per source-ip
accept;
}
}
term dest_hash {
from {
prefix-list-filter clients_v4 exact;
}
then {
load-balance destination-ip-only; ##when clients, then LB per dest-ip
accept;
}
}
term ALL-ELSE {
then {
load-balance per-packet; ### for anything else
accept;
}
}
}
}
routing-options {
forwarding-table {
export pfe_lb_hash;
}
}The following MX Series Router configuration is an example for specific forward and return traffic with CHASH:
policy-options {
policy-statement pfe_consistent_hash { ### Load Balancing mechanism
from {
route-filter 0.0.0.0/0 exact;
}
then {
load-balance consistent-hash;
accept;
}
}
policy-statement pfe_sfw_return_consistent_hash {### Load Balancing mechanism
from {
prefix-list-filter clients_v4 exact;
}
then {
load-balance consistent-hash;
accept;
}
}
policy-statement trust-to-untrust-export { ### Export static + learned BGP routes
term 1 {
from protocol [ bgp static ];
then {
next-hop self;
accept;
}
}
term 2 {
then reject;
}
}
policy-statement untrust-to-trust-export { ### Export static + learned BGP routes
term 1 {
from protocol [ bgp static ];
then {
next-hop self;
accept;
}
}
term 2 {
then reject;
}
}
policy-statement client_sfw_and_nat_pool_prefix_export {
term 1 {
from {
protocol bgp;
route-filter 192.0.2.0/25 exact; ### clients subnet(s)
route-filter 100.64.1.0/24 exact; ### NATPool SRX1 or SRXMNHA pair1
route-filter 100.64.2.0/24 exact; ### NATPool SRX1 or SRXMNHA pair2
route-filter 100.64.3.0/24 exact; ### NATPool SRX1 or SRXMNHA pair3
...
}
then {
next-hop self;
accept;
}
}
term 2 {
then reject;
}
}
}The following MX Series Router configuration is for the routing instance and BGP peering with default GW and the SRX Series Firewall:
routing-instances {
TRUST_VR {
instance-type virtual-router;
routing-options {
autonomous-system 65536;
static {
route 192.0.2.0/25 next-hop 10.40.1.1; ### Internal route toward clients
} ### Or internal BGP peering (not shown)
}
protocols {
bgp {
group MX-TO-SRXS { ### BGP Peering with all SRX (trust)
type external;
import pfe_consistent_hash; ### apply LB CHASH for clients
export trust-to-untrust-export; ### Export learned routes to peers
peer-as 64500;
local-as 65536;
multipath;
bfd-liveness-detection {
minimum-interval 300;
minimum-receive-interval 300;
multiplier 3;
}
neighbor 10.1.1.0; ### PEERING WITH SRX1
neighbor 10.1.1.8; ### PEERING WITH SRX2
... ### ANY OTHER SRX/VSRX
}
}
}
interface ae1.0;
interface ae2.20;
...
}
UNTRUST_VR {
instance-type virtual-router;
routing-options {
autonomous-system 65550;
static {
route 0.0.0.0/0 next-hop 10.80.1.1;### EITHER Ext default GW router
}
}
protocols {
bgp {
group MX-TO-GATEWAY { ### OR External default GW router
type external;
export client_sfw_and_nat_pool_prefix_export; ### Export clients/NAT routes to GW
neighbor 10.80.1.1; ### Peering with GW
peer-as 65551;
local-as 65550;
...
}
group MX-TO-SRXS { ### BGP Peering with all SRX (untrust)
type external;
import pfe_sfw_return_consistent_hash; ### Apply LB per clients dest IP
export untrust-to-trust-export; ### Export learned routes to peers
peer-as 64500;
local-as 65550;
multipath;
bfd-liveness-detection {
minimum-interval 300;
minimum-receive-interval 300;
multiplier 3;
}
neighbor 10.1.1.2; ### PEERING WITH SRX1
neighbor 10.1.1.10; ### PEERING WITH SRX2
... ### ANY OTHER SRX/VSRX
}
}
interface ae...;
...
}
}The following is the sample SRX1 configuration for SFW and CGNAT:
policy-options {
policy-statement trust_export_policy {
term 1 {
from {
protocol bgp;
route-filter 0.0.0.0/0 exact; ### SRX announce 0/0 to MX
}
then {
next-hop self;
accept;
}
}
term 2 {
then reject;
}
}
policy-statement untrust_export_policy {
term 1 {
from {
protocol bgp;
route-filter 192.0.2.0/25 orlonger; ### SRX announce clients to MX
}
then accept;
}
term 2 {
from {
protocol static;
route-filter 100.64.1.0/24 orlonger; ### SRX1 filters NAT POOL1
### route-filter 100.64.2.0/24 orlonger; ### SRX2 filters NAT POOL2
### route-filter 100.64.3.0/24 orlonger; ### SRX3 filters NAT POOL3
}
then accept;
}
term 3 {
then reject;
}
}
}
routing-instances {
VR-1 {
instance-type virtual-router;
routing-options {
static {
route
100.64.1.0/24
discard; ### SRX1 installs NAT POOL1 in table
### route 100.64.2.0/24 discard; ### SRX2 installs NAT POOL2 in table
### route 100.64.3.0/24 discard; ### SRX3 installs NAT POOL3 in table
}
}
protocols {
bgp {
group srx-to-mx1_TRUST {
type external;
export trust_export_policy; ### announces 0.0.0.0/0 to MX trust
local-as 64500;
bfd-liveness-detection {
minimum-interval 300;
minimum-receive-interval 300;
multiplier 3;
}
neighbor 10.1.1.1 { ### SRX1 uses source address 10.1.1.0
peer-as 65536;
}
}
group srx-to-mx1_UNTRUST {
type external;
export untrust_export_policy; ### announces clients and NAT POOLs to MX untrust
local-as 64500;
bfd-liveness-detection {
minimum-interval 300;
minimum-receive-interval 300;
multiplier 3;
}
neighbor 10.1.1.3 { ### SRX1 uses source address 10.1.1.2
peer-as 65550;
}
}
}
}
interface ae1.0; ### Interface assigned to TRUST zone
interface ae1.1; ### Interface assigned to UNTRUST zone
}
}
security {
zones {
security-zone trust {
host-inbound-traffic {
system-services {
ping;
}
protocols {
bfd;
bgp;
}
}
interfaces {
ae1.0;
}
}
security-zone untrust {
screen untrust-screen;
host-inbound-traffic {
system-services {
ping;
}
protocols {
bfd;
bgp;
}
}
interfaces {
ae1.1;
}
}
}
nat {
source {
pool vsrx1_nat_pool {
address {
100.64.1.0/24; ### example NAT POOL SRX1 (or SRX MNHA pair1)
### 100.64.2.0/24; ### example NAT POOL SRX2 (or SRX MNHA pair2)
### 100.64.3.0/24; ### example NAT POOL SRX3 (or SRX MNHA pair3)
}
address-pooling paired;
}
rule-set vsrx1_nat_rule-set {
from zone trust;
to zone untrust;
rule vsrx1_nat_rule1 {
match {
source-address 192.0.2.0/25;
destination-address 0.0.0.0/0;
application any;
}
then {
source-nat {
pool {
vsrx1_nat_pool;
}
}
}
}
}
}
}
policies {
from-zone trust to-zone untrust { ### outbound permit security policy
policy t2u-permit {
match {
source-address any;
destination-address any;
application any;
}
then {
permit;
log {
session-close;
}
}
}
}
from-zone untrust to-zone trust { ### inbound deny security policy
policy u2t-permit {
match {
source-address any;
destination-address any;
application any;
}
then {
deny; ### change to permit if needed
log {
session-close;
}
}
}
}
default-policy {
deny-all;
}
pre-id-default-policy {
then {
log {
session-close;
}
}
}
}
}While running tests, some of the ECMP/CHASH outputs shows route selections:
user@MX> show route table trust-vr.inet.0 0.0.0.0/0 exact active-path extensive
trust-vr.inet.0: 24 destinations, 28 routes (24 active, 0 holddown, 0 hidden)
0.0.0.0/0 (5 entries, 1 announced)
TSI:
KRT in-kernel 0.0.0.0/0 -> {list: 10.1.1.0, 10.1.1.8, 10.1.1.16 Flags source ip load-balance} ### Source-ip LB
*BGP Preference: 170/-101
Next hop type: Router, Next hop index: 0
Address: 0x9dd701c
Next-hop reference count: 2, key opaque handle: 0x0, non-key opaque handle: 0x0
Source: 10.1.1.1
Next hop: 10.1.1.0 via ae1.0
Session Id: 0
Next hop: 10.1.1.8 via ae2.0
Session Id: 0
Next hop: 10.1.1.16 via ae3.0, selected
Session Id: 0
State: <Active Ext LoadBalConsistentHash> ### CHASH used
Local AS: 64500 Peer AS: 65536
Age: 50:43
Validation State: unverified
Task: BGP_64500_65536.10.1.1.0
Announcement bits (2): 0-KRT 1-BGP_Multi_Path
AS path: 64500 65550 I
Accepted Multipath
Localpref: 100
Router ID: 10.1.1.0
Thread: junos-main
user@MX> show route table untrust-vr.inet.0 192.0.2.0/25 exact active-path extensive
untrust-vr.inet.0: 24 destinations, 28 routes (24 active, 0 holddown, 0 hidden)
192.0.2.0/25 (5 entries, 1 announced)
TSI:
KRT in-kernel 192.0.2.0/25 -> {list:10.1.1.2, 10.1.1.10, 10.1.1.18 Flags destination ip load-balance} ### Dest-ip LB
*BGP Preference: 170/-101
Next hop type: Router, Next hop index: 0
Address: 0x9dd821c
Next-hop reference count: 2, key opaque handle: 0x0, non-key opaque handle: 0x0
Source: 10.1.1.1
Next hop: 10.1.1.2 via ae1.1
Session Id: 0
Next hop: 10.1.1.10 via ae2.1
Session Id: 0
Next hop: 10.1.1.18 via ae3.1, selected
Session Id: 0
State: <Active Ext LoadBalConsistentHash> ### CHASH used
Local AS: 64500 Peer AS: 65550
Age: 50:44
Validation State: unverified
Task: BGP_64500_65550.10.1.1.2
Announcement bits (2): 0-KRT 1-BGP_Multi_Path
AS path: 64500 65536 I
Accepted Multipath
Localpref: 100
Router ID: 10.1.1.2
Thread: junos-mainThis configuration is also available in the CSDS configuration example as the same technology and configuration is used for the ECMP CHASH. However, some of the IP addresses or AS may have changed. For more information, see https://www.juniper.net/documentation/us/en/software/connected-security-distributed-services/csds-deploy/topics/example/configure-csds-ecmp-chash-singlemx-standalonesrx-scaledout-nat-statefulfw.html .
Configuration Examples for TLB
Like ECMP CHASH, the trust-vr/untrust-vr are similar in the TLB use case, with BGP peering with the SRX Series Firewalls on each side. However, different configuration is needed for the TLB services, including additional routing-instances, and less policy statements.
Source-hash for forward flow and destination-hash for reverse flow is common for all the ECMP based or TLB based solutions. Consistent hash is used during any next-hop failures where it helps to ensure that the existing sessions on active next-hops are not disturbed and sessions only on the down next-hops are redistributed over other active next-hops. This consistent hash behavior is prebuilt in the TLB solution.
These configuration examples can be used for IPv6.
Following sample configuration shows general load balancing strategy for anything but TLB:
system { ### internal services needed for TLB
processes {
sdk-service enable;
}
}
policy-options {
policy-statement pfe_lb_hash {
term ALL-ELSE {
then {
load-balance per-packet;
accept;
}
}
}
}
routing-options {
forwarding-table {
export pfe_lb_hash;
}
}The following sample MX Series Router configuration shows specific forward and return traffic:
routing-instances {
TRUST_VR {
instance-type virtual-router;
### BGP Peering with next router toward clients
### BGP Peering with each SRX on the TRUST side (similar to ECMP CHASH)
interface lo0.1;
interface ae...;
interface ...; ### other interfaces to/from the internal router
}
UNTRUST_VR {
instance-type virtual-router;
### BGP Peering with next router toward outside
### BGP Peering with each SRX on the UNTRUST side (similar to ECMP CHASH)
interface lo0.2;
interface ae...;
interface ...; ### other interfaces to/from the external router
}
srx_mnha_group_tlb-trust_fi { ### additional forwarding instance for trust redirection
instance-type forwarding;
}
srx_mnha_group_tlb-untrust_fi { ### additional forwarding instance for untrust redirection
instance-type forwarding;
}
}The following sample configuration shows how traffic is redirected to TLB instance using filter-based forwarding (associated with routing-instance srx_mnha_group_tlb-fi):
firewall {
family inet {
filter MX_TLB_LB_TRUST { ### The FBF to redirect traffics to TLB
term NAPT44_tlb_traffic {
from {
source-address {
192.0.2.0/25;
}
}
then {
count SFW44_tlb_forward_traffic;
routing-instance srx_mnha_group_tlb-trust_fi; ### Forwarding instance used by TLB
}
}
term other_traffic {
then {
count other_traffic;
accept;
}
}
}
filter MX_TLB_LB_UNTRUST {
term NAPT44_tlb_traffic {
from {
destination-address {
192.0.2.0/25;
}
}
then {
count SFW44_tlb_return_traffic;
routing-instance srx_mnha_group_tlb-untrust_fi; ### Forwarding instance used by TLB
}
}
term other_traffic {
then {
count other_traffic;
accept;
}
}
}
}
}
interfaces {
### Aggregate and vlan tagging used on AE (not shown here)
ae1 {
unit 0 {
vlan-id 1;
family inet {
filter {
input MX_TLB_LB_TRUST; ### Where forward traffic FBF is applied
}
address 10.1.1.1/31;
}
}
unit 1 {
vlan-id 2;
family inet {
filter {
input MX_TLB_LB_UNTRUST; ### Where return traffic FBF is applied
}
address 10.1.1.3/31;
}
}
}
}The following sample configuration shows interface loopbacks used by TLB for health checking to the SRX Series Firewalls:
interfaces {
lo0 {
unit 1 {
ip-address-owner service-plane; ### not for RE-TLB but used on other MX
description "TLB Health-Check Source IP Addresses for TRUST VR";
family inet {
address 10.0.0.253/32;
}
}
unit 2 {
ip-address-owner service-plane; ### not for RE-TLB but used on other MX
description "TLB Health-Check Source IP Addresses for UNTRUST VR";
family inet {
address 10.0.0.254/32;
}
}
}
}The following sample configuration shows the TLB service part (example with an NAT service here, then only trust side TLB instance is used as NAT Pools are announced for return traffic):
services {
traffic-load-balance {
routing-engine-mode; ### Important for MX304/MX10K to enable TLB
instance tlb_sfw_trust { ### TLB instance for trust forward traffics
interface lo0.1;
client-vrf TRUST_VR;
server-vrf TRUST_VR;
group srx_trust_group {
real-services [ MNHA_SRX1 MNHA_SRX2 ]; ### selected SRXs in that TLB group
routing-instance TRUST_VR;
health-check-interface-subunit 1;
network-monitoring-profile icmp-profile;
}
real-service MNHA_SRX1 { ### Loopback address used on MNHA SRX1 pair
address 10.0.0.1;
}
real-service MNHA_SRX2 { ### Loopback address used on MNHA SRX2 pair
address 10.0.0.2;
}
...
virtual-service srx_trust_vs {
mode direct-server-return;
address 0.0.0.0;
routing-instance srx_mnha_group_tlb-trust_fi; ### Using routes from this VR
group srx_trust_group; ### and sending them to that TLB group
load-balance-method {
hash {
hash-key {
source-ip; ### using source-ip as hash
}
}
}
}
}
instance tlb_sfw_untrust { ### TLB instance for untrust return traffics
interface lo0.2;
client-vrf UNTRUST_VR;
server-vrf UNTRUST_VR;
group srx_untrust_group {
real-services [ UNTRUST_SRX1 UNTRUST_SRX2 ];
routing-instance UNTRUST_VR;
health-check-interface-subunit 2;
network-monitoring-profile icmp-profile;
}
real-service UNTRUST_SRX1 { ### Loopback address used on MNHA SRX1 pair
address 10.0.0.1;
}
real-service UNTRUST_SRX2 { ### Loopback address used on MNHA SRX2 pair
address 10.0.0.2;
}
virtual-service srx_untrust_vs {
mode direct-server-return;
address 0.0.0.0;
routing-instance srx_mnha_group_tlb-untrust_fi; ### Using routes from this VR
group srx_untrust_group; ### and sending them to that TLB group
load-balance-method {
hash {
hash-key {
destination-ip; ### using destination-ip as hash
}
}
}
}
}
}
network-monitoring { ### monitor via icmp, http, tcp, udp, ssl/tls…
profile icmp-profile {
icmp;
probe-interval 2;
failure-retries 3;
recovery-retries 5;
}
}
}The following sample is SRX1 configuration for SFW:
interfaces {
lo0 {
unit 1 {
family inet {
address 10.0.0.1/32;
}
}
}
}
policy-options {
policy-statement trust_export_policy {
term 1 {
from {
protocol direct;
route-filter 10.0.0.1/32 exact; ### announce only local loopback
condition srg_sig_route_exist; ### if MNHA state is true
}
then {
as-path-prepend "64500 64500 64500"; ### Added more route cost
next-hop self;
accept;
}
}
term 2 {
from {
protocol direct;
route-filter 10.0.0.1/32 exact;
}
then {
as-path-prepend 64500; ### Added less route cost
next-hop self;
accept;
}
}
term 3 {
then reject;
}
}
policy-statement untrust_export_policy {
term 1 {
from {
protocol direct;
route-filter 10.0.0.1/32 exact; ### filter on local loopback
condition srg_sig_route_exist; ### if MNHA state is true
}
then {
as-path-prepend "64500 64500 64500"; ### Added more route cost
next-hop self;
accept;
}
}
term 2 {
from {
protocol direct;
route-filter 10.0.0.1/32 exact; ### OR
}
then {
as-path-prepend 64500; ### Added less route cost
next-hop self;
accept;
}
}
term 3 {
then reject;
}
}
}
routing-instances {
VR-1 {
instance-type virtual-router;
protocols {
bgp {
group srx-to-mx1_TRUST {
type external;
export trust_export_policy;
local-as 64500;
bfd-liveness-detection {
minimum-interval 300;
minimum-receive-interval 300;
multiplier 3;
}
neighbor 10.1.1.1 {
peer-as 65536;
}
}
group srx-to-mx1_UNTRUST {
type external;
export untrust_export_policy;
local-as 64500;
bfd-liveness-detection {
minimum-interval 300;
minimum-receive-interval 300;
multiplier 3;
}
neighbor 10.1.1.3 {
peer-as 65550;
}
}
}
}
interface ae1.0; ### Interface assigned to TRUST zone
interface ae1.1; ### Interface assigned to UNTRUST zone
interface lo0.1; ### Interface used for health check from MX TLB
}
}
security {
zones {
security-zone trust {
host-inbound-traffic {
system-services {
ping;
}
protocols {
bfd;
bgp;
}
}
interfaces {
ae1.0;
lo0.1;
}
}
security-zone untrust {
screen untrust-screen;
host-inbound-traffic {
system-services {
ping;
}
protocols {
bfd;
bgp;
}
}
interfaces {
ae1.1;
}
}
}
}While running tests, some output for TLB on the MX Series Router could be seen as the group usage and packets/bytes to each SRX Series Firewalls:
user@MX> show services traffic-load-balance statistics instance tlb_sfw_trust Traffic load balance instance name : tlb_sfw_trust Network monitor RE daemon connection : Established Interface type : Multi services Route hold timer : 180 Active real service count : 2 Total real service count : 2 Traffic load balance virtual svc name : srx_trust_vs IP address : 0.0.0.0 Virtual service mode : Direct Server Return mode Routing instance name : srx_mnha_group_tlb-trust_fi Traffic load balance group name : srx_trust_group Health check interface subunit : 1 Demux Nexthop index : N/A (810) Nexthop index : 814 Up time : 1d 11:19 Total packet sent count : 81689080324 Total byte sent count : 54701197361434 Real service Address Sts Packet Sent Byte Sent Packet Recv Byte Recv MNHA_SRX1 10.0.0.1 UP 40823617372 27336541775865 MNHA_SRX2 10.0.0.2 UP 40865462888 27364655583009 user@MX> show services traffic-load-balance statistics instance tlb_sfw_untrust Traffic load balance instance name : tlb_sfw_untrust Network monitor RE daemon connection : Established Interface type : Multi services Route hold timer : 180 Active real service count : 2 Total real service count : 2 Traffic load balance virtual svc name : srx_trust_vs IP address : 0.0.0.0 Virtual service mode : Direct Server Return mode Routing instance name : srx_mnha_group_tlb-untrust_fi Traffic load balance group name : srx_untrust_group Health check interface subunit : 1 Demux Nexthop index : N/A (812) Nexthop index : 815 Up time : 1d 11:20 Total packet sent count : 62220054022 Total byte sent count : 46245032487920 Real service Address Sts Packet Sent Byte Sent Packet Recv Byte Recv UNTRUST_SRX1 10.0.0.1 UP 31082481387 23096450231084 UNTRUST_SRX2 10.0.0.2 UP 31137572635 23148582256836
Common Configurations for ECMP CHASH and TLB
Some elements of configuration need to be in place for both load balancing methods. The following sample configurations are for TRUST and UNTRUST VR and the peering with each SRX Series Firewalls. It also shows some other less seen configuration elements.
Following is some of the common configurations when using dual MX Series Router topology:
- Both MX Series Router calculate same hash value when both have
same number of next hops.
forwarding-options { enhanced-hash-key { symmetric; } }