Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

Navigation
Guide That Contains This Content
[+] Expand All
[-] Collapse All

    Troubleshooting High CPU Usage

    Problem

    Security policy rules processing has an impact on CPU usage. You can use scio signature statistic tool to understand the CPU utilization used to match particular attack objects or attack groups. You can then take a better informed approach to tuning your security policy.

    Note: There is a slight performance impact to using this feature. Enable this feature while you are investigating high CPU; otherwise, disable it.

    To use the scio signature statistic tool:

    1. Log into the CLI as admin and enter su - to switch to root.
    2. Enter the following command to enable the signature statistic tool:
      [root@defaulthost ~]# scio const -s s0 set sc_enable_sigstat 1
    3. Configure a threshold to target attack objects that use a lot of CPU.

      1. Enter the following command to get started:
        [root@defaulthost ~]# scio const -s s0 set sc_sigstat_nano_secs 1000000

        In this example, the value 1000000 is a filter to log only attack objects that use more than 1,000,000 CPU cycles. The default is 10,000 (0x2710).

      2. Search the log file to examine its contents. Logs are saved in the /usr/idp/device/var/sysinfo/logs directory. Files are named idpengine_num.date.

        Logs have the following format:

        sigmatch stat: [function name] [attack/group: attack_name/group_name] [cpu cycles: cpu cycles consumed]
        for the flow : srcIP:srcPort -> dst IP: dst Port [protocol number]
        

        For example:

        sigmatch stat: [sc_ids_run_packet] [attack: DNS:OVERFLOW:BIN] [cpu cycles: 12462695]
        for the flow : 192.168.1.100:59773 -> 192.168.1.24:53 [17]
        sigmatch stat: [sc_ids_run_stream256] [group: dg_c11e-1f0d0p17s0n00_10] [cpu cycles: 31026226]
        for the flow : 192.168.1.100:1502 -> 192.168.1.24:8080 [6]

        In the above example, note that the attack object DNS:OVERFLOW:BIN and the attack group dg_c11e-1f0d0p17s0n00_10 are using a large number of CPU cycles.

      3. If you get too many matches, you can adjust your filter for sc_sigstat_nano_secs (upwards to narrow the set that matches, downwards to broaden the set that matches).
      4. Make a note of attack objects and groups that generate a lot of matches. You probably do not need to be concerned about occasional matches. You want to identify patterns of high CPU cycles.

        Use the following command syntax to display the attacks contained in the attack group:

        [root@defaulthost ~]# scio policy attacks subscriber policy.set-path detector-path group-name

        For example:

        [root@defaulthost ~]# scio policy attacks s0 /usr/idp/device/state/s0/policy.set /usr/idp/device/lkm/detector.o.gz.v dg_c11e-1f0d0p17s0n00_10

        The following is example output:

        Attacks in the group 'dg_c11e-1f0d0p17s0n00_10' are... [10 attacks]
        
                         APP:UPNP:APPLE-MDNS_1
                         MS-RPC:MESSENGER-OF2_1
                         APP:UPNP:APPLE-MDNS_2
                         RPC:RPC.SADMIND:METHOD-TRAV-UDP
                         APP:UPNP:APPLE-MDNS_3
                         MS-RPC:MESSENGER-OF2_2
                         MS-RPC:MESSENGER-OF
                         MS-RPC:MESSENGER-OF2_3
                         SCAN:CANVAS:LINUXSNMP
                         TROJAN:BACKORIFICE:BOPING-WIN
        
    4. If necessary, use the suggestions in the next section to refine your security policy rules. We recommend creating a separate rule for an attack object or group that has a pattern of high CPU usage. Later, if you encounter an issue with high CPU usage, you have the option of setting the rule action temporarily to Ignore while you investigate the CPU spike.

    Solution

    If you observe high CPU usage, consider the following adjustments to your security policy rules:

    • In general, it is better to create more rules with a few attack objects each than to create few rules with many attack objects in each. Being specific with rules might also facilitate network analysis and troubleshooting.
    • Ensure your rule includes only the attack objects applicable to the service and destination server identified in the rule. Each attack object has a performance cost.
    • Include only the attack objects related to the traffic direction. For example, in most cases you want to exclude server-to-client attacks from rules protecting destination servers and to exclude client-to-server attacks from rules where you monitor server response.

    Published: 2011-05-02