Monitoring and Troubleshooting Security Policies
Monitoring provides a real-time presentation of meaningful data representing the state of access activities on a network. This insight allows you to easily interpret and effect operational conditions. Troubleshooting provides contextual guidance for resolving the access issues on networks. You can then address user concerns and provide resolution in a timely manner.
Understanding Security Alarms
Alarms are triggered when packets are dropped because of a policy violation. A policy violation occurs when a packet matches a reject or deny policy. A policy violation alarm is generated when the system monitors any of the following audited events:
Number of policy violations by a source network identifier within a specified period
Number of policy violations to a destination network identifier within a specified period
Number of policy violations to an application within a specified period
Policy rule or group of rule violations within a specified period
There are four types of alarms corresponding to these four events. The alarms are based on source IP, destination IP, application, and policy.
When a packet encounters a reject or deny policy, the policy violation counters for all enabled types of alarm are increased. When any counter reaches the specified threshold within a specified period, an alarm is generated. After a specified period, the policy violation counter is reset and reused to start another counting cycle.
To view the alarm information, run the show security alarms command. The violation count and the alarm do not persist across
system reboots. After a reboot, the violation count resets to zero
and the alarm is cleared from the alarm queue.
After taking appropriate actions, you can clear the alarm. The
alarm remains in the queue until you clear it (or until you reboot
the device). To clear the alarm, run the clear security alarms command. After you clear the alarm, a subsequent series of flow
policy violations can cause a new alarm to be raised.
See Also
Example: Generating a Security Alarm in Response to Policy Violations
This example shows how to configure the device to generate a system alarm when a policy violation occurs. By default, no alarm is raised when a policy violation occurs.
Requirements
No special configuration beyond device initialization is required before configuring this feature.
Overview
In this example, you configure an alarm to be raised when:
The application size is 10240 units.
The source IP violation exceeds 1000 within 20 seconds.
The destination IP violations exceeds 1000 within 10 seconds.
The policy match violation exceeds 100, with a size of 100 units.
Configuration
Procedure
CLI Quick Configuration
To quickly configure this example, copy the
following commands, paste them into a text file, remove any line breaks,
change any details necessary to match your network configuration,
copy and paste the commands into the CLI at the [edit] hierarchy
level , and then enter commit from configuration mode.
set security alarms potential-violation policy application size 10240 set security alarms potential-violation policy source-ip threshold 1000 duration 20 set security alarms potential-violation policy destination-ip threshold 1000 duration 10 set security alarms potential-violation policy policy-match threshold 100 size 100
Step-by-Step Procedure
The following example requires you to navigate various levels in the configuration hierarchy. For instructions on how to do that, see Using the CLI Editor in Configuration Mode in the CLI User Guide.
To configure policy violation alarms:
Enable security alarms.
[edit] user@host# edit security alarms
Specify that an alarm should be raised when an application violation occurs.
[edit security alarms potential-violation policy] user@host# set application size 10240
Specify that an alarm should be raised when a source IP violation occurs.
[edit security alarms potential-violation policy] user@host# set source-ip threshold 1000 duration 20
Specify that an alarm should be raised when a destination IP violation occurs.
[edit security alarms potential-violation policy] user@host# set destination-ip threshold 1000 duration 10
Specify that an alarm should be raised when a policy match violation occurs.
[edit security alarms potential-violation policy] user@host# set policy-match threshold 100 size 100
Results
From configuration mode, confirm your configuration
by entering the show security alarms command. If the output
does not display the intended configuration, repeat the configuration
instructions in this example to correct it.
policy {
source-ip {
threshold 1000;
duration 20;
}
destination-ip {
threshold 1000;
duration 10;
}
application {
size 10240;
}
policy-match {
threshold 100;
size 100;
}
}
If you are done configuring the device, enter commit from configuration mode.
Verification
To confirm that the configuration is working
properly, from operational mode, enter the show security alarms command.
Matching Security Policies
The show security match-policies command allows you
to troubleshoot traffic problems using the match criteria: source
port, destination port, source IP address, destination IP address,
and protocol. For example, if your traffic is not passing because
either an appropriate policy is not configured or the match criteria
is incorrect, the show security match-policies command
allows you to work offline and identify where the problem actually
exists. It uses the search engine to identify the problem and thus
enables you to use the appropriate match policy for the traffic.
The result-count option specifies how many policies
to display. The first enabled policy in the list is the policy that
is applied to all matching traffic. Other policies below it are “shadowed”
by the first and are never encountered by matching traffic.
The show security match-policies command is
applicable only to security policies; IDP policies are not supported.
Example 1: show security match-policies
user@host> show security match-policies from-zone z1 to-zone z2 source-ip 10.10.10.1 destination-ip 192.0.2.1 source-port 1 destination-port 21 protocol tcp
Policy: p1, action-type: permit, State: enabled, Index: 4
Sequence number: 1
From zone: z1, To zone: z2
Source addresses:
a2: 203.0.113.1/25
a3: 10.10.10.1/32
Destination addresses:
d2: 203.0.113.129/25
d3: 192.0.2.1/24
Application: junos-ftp
IP protocol: tcp, ALG: ftp, Inactivity timeout: 1800
Source port range: [0-0]
Destination port range: [21-21]
Example 2: Using the result-count Option
By default, the output list contains the policy that will be
applied to traffic with the specified characteristics. To list more
than one policy that match the criteria, use the result-count option. The first policy listed is always the policy that will be
applied to matching traffic. If the result-count value
is from 2 to 16, the output includes all policies that match the criteria
up to the specified result-count. All policies listed after
the first are “shadowed” by the first policy and are never
applied to matching traffic.
Use this option to test the positioning of a new policy or to troubleshoot a policy that is not applied as expected for particular traffic.
In the following example, the traffic criteria matches two policies.
The first policy listed, p1, contains the action applied
to the traffic. Policy p15 is shadowed by the first policy,
and its action, therefore, will not be applied to matching traffic.
user@host> show security match-policies from-zone zone-A to-zone zone-B source-ip 10.10.10.1 destination-ip 192.0.2.1 source-port 1004 destination-port 80 protocol tcp result-count 5
Policy: p1, action-type: permit, State: enabled, Index: 4
Sequence number: 1
From zone: zone-A, To zone: zone-B
Source addresses:
sa1: 10.10.0.0/16
Destination addresses:
da5: 192.0.2.0/24
Application: any
IP protocol: 1, ALG: 0, Inactivity timeout: 0
Source port range: [1000-1030]
Destination port range: [80-80]
Policy: p15, action-type: deny, State: enabled, Index: 18
Sequence number: 15
From zone: zone-A, To zone: zone-B
Source addresses:
sa11: 10.10.10.1/32
Destination addresses:
da15: 192.0.2.5/24
Application: any
IP protocol: 1, ALG: 0, Inactivity timeout: 0
Source port range: [1000-1030]
Destination port range: [80-80]
See Also
Tracking Policy Hit Counts
Use the show security policies hit-count command
to display the utility rate of security policies according to the
number of hits they receive. You can use this feature to determine
which policies are being used on the device, and how frequently they
are used. Depending on the command options that you choose, the number
of hits can be listed without an order or sorted in either ascending
or descending order, and they can be restricted to the number of hits
that fall above or below a specific count or within a range. Data
is shown for all zones associated with the policies or named zones.
Checking Memory Usage on SRX Series Devices
You can isolate memory issues by comparing memory values before and after policy configurations.
Memory for flow entities such as policies, zones, or addresses on SRX1400, SRX1500, SRX3400, SRX3600, SRX4100, SRX4200, SRX4600, SRX5400, SRX5600, and SRX5800 devices (depending on the Junos OS release in your installation) is dynamically allocated. However, certain practices can help monitor the current memory usage on the device and optimize parameters to better size system configuration, especially during policy implementation.
To check memory usage:
Use the
show chassis routing-enginecommand to check overall Routing Engine (RE) memory usage. The following output from this command shows memory utilization at 39 percent:user@host#
show chassis routing-engineRouting Engine status: Slot 0: Current state Master Election priority Master (default) DRAM 1024 MB Memory utilization 39 percent CPU utilization: User 0 percent Background 0 percent Kernel 2 percent Interrupt 0 percent Idle 97 percent Model RE-PPC-1200-A Start time 2011-07-09 19:19:49 PDT Uptime 37 days, 15 hours, 44 minutes, 13 seconds Last reboot reason 0x3:power cycle/failure watchdog Load averages: 1 minute 5 minute 15 minute 0.22 0.16 0.07Use the
show system processes extensivecommand to acquire information on the processes running on the Routing Engine.Use the
find nsdoption in theshow system processes extensivecommand to see direct usage on the Network Security Daemon (NSD) with its total memory in use as 10 megabytes and CPU utilization of 0 percent.user@host# show system processes extensive | find nsd 1182 root 1 96 0 10976K 5676K select 2:08 0.00% nsd 1191 root 4 4 0 8724K 3764K select 1:57 0.00% slbd 1169 root 1 96 0 8096K 3520K select 1:51 0.00% jsrpd 1200 root 1 4 0 0K 16K peer_s 1:10 0.00% peer proxy 1144 root 1 96 0 9616K 3528K select 1:08 0.00% lacpd 1138 root 1 96 0 6488K 2932K select 1:02 0.00% ppmd 1130 root 1 96 0 7204K 2208K select 1:02 0.00% craftd 1163 root 1 96 0 16928K 5188K select 0:58 0.00% cosd 1196 root 1 4 0 0K 16K peer_s 0:54 0.00% peer proxy 47 root 1 -16 0 0K 16K sdflus 0:54 0.00% softdepflush 1151 root 1 96 0 15516K 9580K select 0:53 0.00% appidd 900 root 1 96 0 5984K 2876K select 0:41 0.00% eventd
Check the configuration file size. Save your configuration file with a unique name before exiting the CLI. Then, enter the
ls -1 filenamecommand from the shell prompt in the UNIX-level shell to check the file size as shown in the following sample output:user@host> start shell % ls -l config -rw-r--r-- 1 remote staff 12681 Feb 15 00:43 config
See Also
Monitor Security Policy Statistics
Purpose
Monitor and record traffic that Junos OS permits or denies based on previously configured policies.
Action
To monitor traffic, enable the count and log options.
Count—Configurable in an individual policy. If count is enabled, statistics are collected for sessions that enter the device for a given policy, and for the number of packets and bytes that pass through the device in both directions for a given policy. For counts (only for packets and bytes), you can specify that alarms be generated whenever the traffic exceeds specified thresholds. See count (Security Policies).
Log—Logging capability can be enabled with security policies during session initialization (session-init) or session close (session-close) stage. See log (Security Policies).
To view logs from denied connections, enable log on session-init.
To log sessions after their conclusion/tear-down, enable log on session-close.
Session log is enabled at real time in the flow code which impacts the user performance. If both session-close and session-init are enabled, performance is further degraded as compared to enabling session-init only.
For details about information collected for session logs, see Information Provided in Session Log Entries for SRX Series Services Gateways.
Verifying Shadow Policies
- Verifying All Shadow Policies
- Verifying a Policy Shadows One or More Policies
- Verifying a Policy Is Shadowed by One or More Policies
Verifying All Shadow Policies
Purpose
Verify all the policies that shadows one or more policies.
Action
From the operational mode, enter the following commands:
For logical systems, enter the
show security shadow-policies logical-system lsys-name from-zone from-zone-name to-zone to-zone-namecommand.For global policies, enter the
show security shadow-policies logical-system lsys-name globalcommand.
root@host> show security shadow-policies from-zone zone-a to-zone zone-b
Policies Shadowed policies
P1 P3
P1 P4
P2 P5
Meaning
The output displays the list of all policies that shadows other policies. In this example, P1 policy shadows P3 and P4 policies and P2 policy shadows P5 policy.
Verifying a Policy Shadows One or More Policies
Purpose
Verify if a given policy shadows one or more policies positioned after it.
Action
From the operational mode, enter the following commands:
For logical systems, enter the
show security shadow-policies logical-system lsys-name from-zone from-zone-name to-zone to-zone-name policy policy-namecommand.For global policies, enter the
show security shadow-policies logical-system lsys-name global policy policy-namecommand.
root@host> show security shadow-policies from-zone zone-a to-zone zone-b policy P1
Policies Shadowed policies
P1 P3
P1 P4
Meaning
The output displays all the policies that are shadowed by the given policy. In this example, P1 policy shadows P3 and P4 policies.
Verifying a Policy Is Shadowed by One or More Policies
Purpose
Verify if a given policy is shadowed by one or more positioned before it.
Action
From the operational mode, enter the following commands:
For logical systems, enter the
show security shadow-policies logical-system lsys-name from-zone from-zone-name to-zone to-zone-name policy policy-name reversecommand.For global policies, enter the
show security shadow-policies logical-system lsys-name global policy policy-name reversecommand.
root@host> show security shadow-policies from-zone zone-a to-zone zone-b policy P4 reverse
Policies Shadowed policies
P1 P4
Meaning
The output displays the given policy shadowed by one or more policies. In this example, P4 policy is shadowed by P1 policy.
Troubleshooting Security Policies
- Synchronizing Policies Between Routing Engine and Packet Forwarding Engine
- Checking a Security Policy Commit Failure
- Verifying a Security Policy Commit
- Debugging Policy Lookup
Synchronizing Policies Between Routing Engine and Packet Forwarding Engine
Problem
Description
Security policies are stored in the routing engine and the packet forwarding engine. Security policies are pushed from the Routing Engine to the Packet Forwarding Engine when you commit configurations. If the security policies on the Routing Engine are out of sync with the Packet Forwarding Engine, the commit of a configuration fails. Core dump files may be generated if the commit is tried repeatedly. The out of sync can be due to:
A policy message from Routing Engine to the Packet Forwarding Engine is lost in transit.
An error with the routing engine, such as a reused policy UID.
Environment
The policies in the Routing Engine and Packet Forwarding Engine must be in sync for the configuration to be committed. However, under certain circumstances, policies in the Routing Engine and the Packet Forwarding Engine might be out of sync, which causes the commit to fail.
Symptoms
When the policy configurations are modified
and the policies are out of sync, the following error message displays
- error: Warning: policy might be out of sync between
RE and PFE <SPU-name(s)> Please request security
policies check/resync.
Solution
Use the show security policies checksum command to display the security policy checksum value and use the request security policies resync command to synchronize the
configuration of security policies in the Routing Engine and Packet
Forwarding Engine, if the security policies are out of sync.
See Also
Checking a Security Policy Commit Failure
Problem
Description
Most policy configuration failures occur during a commit or runtime.
Commit failures are reported directly on the CLI when you execute the CLI command commit-check in configuration mode. These errors are configuration errors, and you cannot commit the configuration without fixing these errors.
Solution
To fix these errors, do the following:
Review your configuration data.
Open the file /var/log/nsd_chk_only. This file is overwritten each time you perform a commit check and contains detailed failure information.
Verifying a Security Policy Commit
Problem
Description
Upon performing a policy configuration commit, if you notice that the system behavior is incorrect, use the following steps to troubleshoot this problem:
Solution
Operational show Commands—Execute the operational commands for security policies and verify that the information shown in the output is consistent with what you expected. If not, the configuration needs to be changed appropriately.
Traceoptions—Set the
traceoptionscommand in your policy configuration. The flags under this hierarchy can be selected as per user analysis of theshowcommand output. If you cannot determine what flag to use, the flag optionallcan be used to capture all trace logs.user@host#
set security policies traceoptions <flag all>
You can also configure an optional filename to capture the logs.
user@host# set security policies traceoptions <filename>
If you specified a filename in the trace options, you can look in the /var/log/<filename> for the log file to ascertain if any errors were reported in the file. (If you did not specify a filename, the default filename is eventd.) The error messages indicate the place of failure and the appropriate reason.
After configuring the trace options, you must recommit the configuration change that caused the incorrect system behavior.
Debugging Policy Lookup
Problem
Description
When you have the correct
configuration, but some traffic was incorrectly dropped or permitted,
you can enable the lookup flag in the security policies
traceoptions. The lookup flag logs the lookup related traces
in the trace file.
Solution
user@host# set security policies traceoptions <flag lookup>
High Availability (HA) Synchronization of Address Name Resolving Cache
The Network security process (NSD) restarts when system reboots, HA failover happens, or if the process crashes. During this time, if there are large number of domain name addresses configured in the security polices, SRX Series Firewalls attempt to send requests to DNS server to get all resolved IP addresses. A high amount of system resources are consumed when a large number of DNS queries and responses are exchanged. So, SRX Series Firewalls are unable to obtain a response from the DNS server and the address of a hostname in an address book entry might fail to resolve correctly. This can cause traffic to drop as no security policy or session match is found. The new enhancement on SRX Series Firewalls addresses this problem by caching the DNS query results into a local DNS cache file and periodically synchronizing the DNS cache file from HA primary node to HA backup node. The DNS cache files stores IP addresses, domain name, and TTL values. After the HA failover, the previous backup node becomes primary node. Since all DNS cache results are available on new primary node, security policy processing continues and pass-through traffic is allowed as per the policy rules.
Starting in Junos OS Release 19.3R1, the policy DNS cache memory is synchronized into one local DNS cache file on the HA active node and is copied to the HA backup node to suppress DNS queries or responses during NSD restart.
The following steps are performed for the synchronization to take place:
The policy DNS cache memory is synchronized into one local policy DNS cache file located at the /var/db/policy_dns_cache path every 30 seconds if the policy DNS cache memory content has changed during this period.
The local DNS cache file is synchronized from the HA primary node to HA backup node immediately after the local DNS cache file has been updated in Step 1.
The synchronization includes the following content:
Domain name
IPv4 address list and its TTL (time to live)
IPv6 address list and its TTL
When NSD restarts, it reads and parses the local DNS cache file and imports all cache entries into memory. The synchronization ensures that DNS queries are suppressed during an NSD restart. NSD restarts on new primary node during HA failover as the resolved IP addresses for domain names already exist in DNS cache memory when reading policies configurations. Therefore, new pass-through traffic is allowed as per the security policy after HA failover because all resolved IP addresses for domain names exist inside policies on new primary node’s Routing Engine and Packet Forwarding Engine.