Event Processing Performance
Your JSA configuration might impact the event processing pipeline.
Event processing can be affected by DSM extensions, custom properties, rule tests, and global views. Event parsing and the custom rules engine automatically detect dropped events, run self-monitoring diagnostics, and report which DSM extensions, rules, and properties are slow.
Non-optimized Custom Properties
Custom properties are marked as optimized when they are regularly used for JSA rules or for searching and filtering.
Non-optimized custom properties are parsed by the system, which affects search speeds and the loading rate of the web browser.
Rule Tests That Impact Performance
Rules that test for regular expressions in an event payload affect JSA performance because they search the entire payload.
Before you add a payload test to a rule, use rule filters to reduce the number of events. For example, when you search for a specific message in the active directory logs, apply the following filters to the rule:
Log source type filter
Log source group or specific log source filter
An optional source IP address filter
The Host with port open test can impact performance because it compares passive and active ports with the events and flows that are received by JSA. Before you use the test, do a bidirectional check to ensure that the host responds to the communication request.
A saved search that is grouped by multiple fields generates a global view that has many unique entries. As the volume of data increases, disk usage, processing times, and search performance can be impacted.
To prevent increasing the volume of data, only aggregate searches on necessary fields. You can reduce the impact on the accumulator by adding a filter to your search criteria.
Identifying DSM and Optimized Custom Property Issues
To help you troubleshoot performance degradation, identify issues with any DSM extensions that were recently installed or custom property that was recently enabled.
A DSM extension creates custom parsing methods by using regex pattern matching to extract event data from unsupported or incomplete log sources. Optimized custom properties use regular expression patterns to extract data from events as they are parsed.
The regex patterns that are used in your DSM extension or optimized custom property can impact event processing in JSA. Inefficient regular expressions can incorrectly route data directly to storage, degrade JSA performance, and affect event processing.
DSM and optimized custom property issues can cause the following system notification:
Performance degradation has been detected
in the event pipeline. Events were routed directly to storage.
- Disable any DSM extension or custom property that is recently installed or enabled.
- Choose one of the following options:
If JSA stops dropping events and you receive a system notification, then review your DSM extensions or custom properties to identify and improve the inefficient regex patterns.
If JSA continues dropping events, then multiple DSM extensions or custom properties might be causing a problem with the event pipeline.
- Use SSH to log in to the JSA Event
Processor that is dropping events and type the following command:
/opt/qradar/support/threadTop.sh –p 7777
The command displays the data processing engine activity. The following table describes the columns in the output:
Table 1: Data Processing Engine Columns
Port or process.
- If parser threads run longer than 1500 milliseconds, review
the Java thread stacks by typing the following command:
/opt/qradar/support/threadTop.sh -p 7777 -s -e ".*Event Parser.*" | less
If the Java thread stack contains
java.util.regex.Pattern$Curly.match, then the performance degradation might be caused by your expensive
DSM extensions or custom properties. For more information, see Expensive
DSM extensions were found in Log and Log Source Notifications for JSA Appliances or Expensive
custom properties found in Event and Flow Notifications for JSA Appliances.
If the Java thread stack doesn't have expensive regular expressions, then your DSM extensions or custom properties might have parsing issues. For more information, see the parsing issues topic in the Configuring DSMs Guide.