Flow Distribution and Packet-Ordering
This topic describes about the load distribution and the packet ordering on SRX5000 Line devices.
Understanding Load Distribution in SRX5000 Line Devices
The load distribution algorithm, which is supported on the SRX5800, SRX5600, and SRX5400 devices, is adjusted based on session capacity and processing power. (Actual platform support depends on the Junos OS release in your installation.)
Hash-based session distribution uses a hash table. The SPU session weight table is used to assign an SPU ID to each hash index in the session distribution hash table. This way, the number of sessions created on each SPU using hash-based distribution is proportional to the SPU's weight in the SPU session weight table. Each NPU also keeps an identical SPU session weight table and session distribution hash table that it uses to select an SPU to forward packets that do not match an NPU session.
In the event of a SPU failure, the Routing Engine will reset all of the cards on the dataplane including IOCs and NPCs in order to maintain hash table consistency for session distribution.
In hash-based session distribution, weights are based on session capacity. We recommend the hash session distribution mode when high session capacity is required.
Load distribution on SRX5000 line devices is always hash-based.
Insertion and removal of SPCs causes recalculation of the SPU session weight table at central point initialization time because the chassis must reboot after insertion.
Starting in Junos OS Release 15.1X49-D30, the central point architecture is enhanced to handle higher concurrent sessions and connections per second (cps) for the SRX5000 line device.
The central point architecture enhancements prevent data packets from going through the central point by offloading traffic management to SPUs. The system session capacity is extended, as the session limit on the central point is removed.
- Calculating SPU ID
- Hash-Based Forwarding on the SRX5K-MPC, SRX5K-MPC3-40G10G (IOC3), and the SRX5K-MPC3-100G10G (IOC3)
Calculating SPU ID
The SPU ID for a device equipped with SRX3K-SPC-1-10-40, SRX5K-SPC-2-10-40, or SRX5K-SPC3 Services Processing Card (SPC) is calculated as follows:
SPU ID = (FPC ID X 4) + PIC ID
The SRX3K-SPC-1-10-40, SRX5K-SPC-2-10-40, and SRX5K-SPC3 contains two PICs per card, four PICs per card (FPC), and two PICs per card respectively. For example, a device contains 2 cards in slot 1 (FPC ID 0) and slot 2 (FPC ID 1), the expected SPU IDs are as follows:
For SPC1: (0, 1) and (4, 5), total 4 SPUs in 2 cards.
For SPC2: (0, 1, 2, 3) and (4, 5, 6, 7), total 8 SPUs in 2 cards.
For SPC3: (0, 1) and (4, 5), total 4 SPUs in 2 cards.
For FPC1 (the second card) and PIC1 (the second PIC in the card), the SPU ID is calculated as:
SPU ID = (FPC ID X 4) + PIC ID
= (1 X 4) + 1
= 4 + 1
= 5Use this convention while referring the SPU ID for CLI and SNMP.
Hash-Based Forwarding on the SRX5K-MPC, SRX5K-MPC3-40G10G (IOC3), and the SRX5K-MPC3-100G10G (IOC3)
On these SRX Series Firewalls, a packet goes through a series of events involving different components as it progresses from ingress to egress processing. With the datapath packet forwarding feature, you can obtain quick delivery of I/O traffic over the SRX 5000 line of devices.
The SRX5K-MPC, SRX5K-MPC3-40G10G (IOC3), and SRX5K-MPC3-100G10G (IOC3) are interface cards supported on the SRX5400, SRX5600, and SRX5800 devices. The Modular Port Concentrator (MPC) provides load-balancing services for Services Processing Units (SPUs) by using the hash-based forwarding method.
In hash-based forwarding, the packet might be forwarded by the MPC to a selected SPU (DCP) instead of the central point. This approach enhances session scaling and prevents overloading of the central point.
Hash value calculation involves the following steps:
For IPv4 packets, the hash-based forwarding module generates the hash value based on Layer 3 and Layer 4 information, depending on different Layer 4 protocol types.
For Stream Control Transmission Protocol (SCTP), TCP, UDP, Authentication Header (AH), edge service provider (ESP), and Internet Control Message Protocol (ICMP) protocols, the hash module utilizes Layer 4 information to generate the hash value. For any other protocols, only Layer 3 information is used in hash generation.
For IPv4 fragment packets, the hash value is calculated using only the Layer 3 information. This also applies to the first fragment of the packet.
For non-IP packets, the hash-based forwarding module uses the Layer 2 information to calculate the hash value.
Once a hash value is calculated according to the packet's Layer 2, Layer 3, or Layer 4 information, an SPU ID is assigned to each hash index in the session distribution hash table.
The SRX5K-MPC (IOC2), SRX5K-MPC3-40G10G (IOC3), and SRX5K-MPC3-100G10G (IOC3) can only be used on SRX5400, SRX5600, and SRX5800 devices that are configured for hash-based session distribution.
When the hash-based session distribution mode is enabled, the system changes its behavior to high-session-capacity-based mode when the SRX5K-MPC, SRX5K-MPC3-40G10G (IOC3), and SRX5K-MPC3-100G10G (IOC3) are installed on the device.
On SRX5000 line devices with an SRX5K-MPC, SRX5K-MPC3-40G10G (IOC3), or SRX5K-MPC3-100G10G (IOC3) installed, during a system or an SPU reboot, when the hash-based session distribution mode is enabled, traffic will pass only when all SPUs are up after the reboot.
The MPCs on the IOC3 provide load-balancing services for SPUs by performing hash-based datapath packet forwarding to interconnect with all existing IOCs and SPCs.
The IOC3 processes ingress and egress packets. The IOC3 parses the ingress packet and sends it to the SPU for further security processing, including flow session lookup, zone and policy check, VPN, ALG, and so on.
The IOC3 manages packet data memory and fabric queuing for packet lookup and encapsulation functions.
Starting with Junos OS Release 15.1X49-D10 and Junos OS Release 17.3R1, hash-based session distribution is the default mode for the SRX5400, SRX5600, and SRX5800 devices. Selection of hash keys depends on application protocols.
Starting with Junos OS Release 17.4R1, traffic is hashed and distributed to different SPUs by the IOC, based on a hash-based session distribution algorithm. This enhancement provides an even hash distribution among all SPUs by using a larger fixed-length hash table. In earlier Junos OS releases, the traffic distribution was uneven among all SPUs due to a fixed-length hash table.
The IOC3 sets up a security flow table (IPv4 and IPv6) including key, result table, and packet memory.
The following functions are provided with the flow table:
Flow lookup
Flow insertion and deletion
Security flow aging out
Security flow statistics
Understanding Packet-Ordering Function on SRX5000 Line Devices
The packet-ordering function, which is supported on the SRX5400, SRX5600, and SRX5800, devices and vSRX Virtual Firewall, improves the performance of the device by activating the built-in packet-ordering function of the Packet Ordering Engine on the XLP processor on the application central point.
Two types of the packet ordering modes are supported: hardware and software.
If the packet-ordering function is set to hardware, the load-balancing thread (LBT) and the packet-ordering thread (POT) are offloaded to the packet ordering engine and resources are freed to perform packet processing. If the packet-ordering function is set to software, the load-balancing thread (LBT) and the packet-ordering thread (POT) are running on the SPU. By default, packet-ordering mode using the Packet Ordering Engine (hardware) is enabled on the device. You can disable it with a configuration change that requires a reboot.
The flow thread receives the packets, processes them, and sends or drops them. For packets that require no ordering, the flow thread notifies the Network Acceleration Engine (NAE) egress to send or drop the packets. For packets that require ordering, the flow thread notifies the Packet Ordering Engine to dequeue the packets from the ordering list and to send or drop the packets in order.
Changing Packet-Ordering Mode on SRX5000 Line Devices
The packet-ordering functionality using the Packet Ordering Engine is supported on SRX5400, SRX5800 and SRX5600 devices with next-generation SPCs. (Platform support depends on the Junos OS release in your installation.) By default, packet-ordering mode using the Packet Ordering Engine is enabled. To disable the packet-ordering functionality using the Packet Ordering Engine, you must update the packet-ordering mode on the device.
The following packet ordering modes are supported:
software—Disables the packet-ordering mode using the Packet Ordering Engine.
hardware—Enables the packet-ordering mode using the Packet Ordering Engine. This is the default option.
To disable the packet-ordering mode using the Packet Ordering Engine:
Enter the following command at the CLI configuration prompt to specify the packet-ordering mode.
[edit] user@host# set security forwarding-process application-services packet-ordering-mode software
Use the
show security forwarding-processcommand to review your configuration.[edit] user@host# show security forwarding-process application-services{ packet-ordering-mode software; }Check your changes to the configuration before committing.
[edit] user@host# commit check
warning: System packet ordering mode changed, reboot is required to take effect. If you have deployed a cluster, be sure to reboot all nodes. configuration check succeeds
Commit the configuration.
[edit] user@host# commit
warning: System packet ordering mode changed, reboot is required to take effect. If you have deployed a cluster, be sure to reboot all nodes. commit complete
Reboot the device at an appropriate time.
Use the
show security flow statuscommand to verify the packet-ordering mode.user@host> show security flow statusFlow forwarding mode: Inet forwarding mode: flow based Inet6 forwarding mode: drop MPLS forwarding mode: drop ISO forwarding mode: drop Flow trace status Flow tracing status: off Flow session distribution Distribution mode: RR-based Flow packet orderingOrdering mode: Software (reboot needed to change to Software)
Understanding Session Distribution on SRX5000 Line Devices in Adaptive Mode
Starting in Junos OS Release 15.1X49-D30 and Junos OS Release 17.3R1, adaptive mode session distribution was replaced by enhancements to the central point architecture.
Adaptive mode session distribution is implemented on the SRX5000 line devices running in mixed mode prior to Junos OS Release 15.1X49-D30 and Junos OS Release 17.1R1. Adaptive mode session distribution maximizes use of system resources by taking into account a Services Processing Unit’s (SPU) capacity and its available resources. It is enabled only on SRX5000 line devices running in XLR/XLP mixed mode, that is in chassis deployments in which different types of SPUs are used in different combinations. If an SRX5800, SRX5600, or SRX5400 device contains a mix of next-generation services processing cards (SPCs) and existing SPCs, then adaptive mode session distribution is assumed as the default. For SRX5000 line devices not running in mixed mode, hash-based load balancing is the default.
A Services Processing Card (SPC) contains one or more SPUs each of which processes the packets of a flow according to the security features and other services configured for sessions distributed to it by the central point (CP). An SPU’s CPU load changes from time to time. To fully utilize changing available capacity and adapt session distribution accordingly, in adaptive mode the system assigns a weight to all SPUs dynamically. It is the weight of the SPUs that determine the session distribution.
Each SPU sends its CPU usage information to the central point (CP) periodically. The central point checks these values, calculates the weight every 1 second, and distributes the sessions in such a way as to maximize overall system performance. In other words, In adaptive mode, session distribution is based on a dynamic weighted assignment system that is calculated in real time allowing for full capacity utilization of the CPUs of all SPUs, regardless of their type.
It is the dynamic calculation of weights that distinguishes adaptive mode session distribution from weighted round-robin (WRR) session distribution. While WRR differentiates SPUs and their CPU capacity by calculating and assigning weights to the different types of SPUs, the calculation and assignment is static, that is, it is done only once, at initialization. Adaptive mode improves on the fixed ratio session distribution process of WRR. WRR leads to underutilization of system resources because session processing limits are set based only on the type of SPU and its CPU capacity, not taking into account its available processing power.
For adaptive mode session distribution, the following formula is used to calculate the weight assigned to an SPU:
Wi = Sum(W1-n)*Ci*Si/Sum(C1-n*S1-n)
Where:
Wi— weight assigned to the SPU.Sum(W1-n)— Total weight of system. This values is constant.n—total number of SPUs.Ci—available CPU computational power of the SPU.Si—available session capacity of SPU.
In adaptive mode, when the CPU usage on one SPU is high, fewer sessions are distributed to that SPU. The following examples explains the calculation.
Consider a device with two SPUs. Each SPU’s session capacity is 1 million.
For a certain time:
When SPU1 has 500,000 sessions on it, CPU usage of it is 10 percent:
Available CPU capacity of SPU1 (C1) = 1-10 percent = 90 (percent).
Available session capacity of SPU1 (S1) = 1-500,000/1M = 50 (percent).
When SPU2 has 400,000 sessions on it, CPU usage of it is 20 percent:
Available capacity of SPU2 (C2)= 1-20 percent= 80 (percent).
Available session capacity of SPU2 (S2)= 1-400,000/1M= 60 (percent).
If the weight of the whole system is 100, the separate weight values for each SPU are:
Weight of SPU1 (W1) = 100*90*50/(50*90+80*60) = 48
Weight of SPU2 (W2) = 100*80*60/(50*90+80*60) = 52
For the incoming sessions, 48 percent of session are allocated to SPU1 while 52 percent of packets are allocated to SPU2.
The weighted numbers might take effect on the system within a short period before the central point checks the runtime usage information and adjusts the weights to a new value.