Create a New Log Parser

Use the New Log Parser page to create your own log parser by providing sample logs. You can build your own parser by mapping fields in your sample logs to Security Director Insights event fields, indicating which types of events will generate an incident.

Procedure

To create a new log parser:

  1. Select Configure>Insights>Log Parsers.

    The Log Parsers page appears.

  2. Select the Create icon (plus).

    The New Log Parser page appears.

  3. Complete the configuration according to the guidelines provided in Table 321.
  4. Click Finish and you are presented with the results of your flexible log parser as they are applied to the sample logs provided.

    Review this carefully to determine whether your mapping, filtering, and assignment conditions are as expected.

Table 321: Fields on the Log Parsers Page

 

Setting

Guideline

Create/Edit Parser

Name

Enter a unique and descriptive name for the log parser.

Description

Enter a description for the log parser.

Parse Log File

Raw Log

Upload the raw log file by browsing to it or place it into the field provided below the Browse button.

Log File Format

Specify the format of the sample log file. The available options are:

  • XML

  • JSON

  • CSV

  • Others

CSV Headers

(If the log file format is ’CSV’)

If your log file is in CSV format, you may provide a comma delimited list of field names in this field. If the CSV headers are not provided, the fields will be named as csv[N], where N is the field position.

Grok Pattern

(If the log file format is ’others’

If you select log file format as ’Others’, you must supply a grok pattern for the log file. A grok pattern may consist of one or more lines. The grok pattern line beginning with "LOGPATTERN" is the pattern which will be applied to the logs. A grok pattern must include a pattern named LOGPATTERN, otherwise the parser will not have any pattern to use.

Field Mapping

Mapped Fields

Unmapped Fields

In the Unmapped Field section, you select one or more check boxes beside a field that has been parsed from your log file. On the right side (Insights Fields), you select a predefined description of the field from the provided options. For example, you might map a field in the Parsed Fields called “interface_ip” to a Insights Field on the right side called “Endpoint IP”.

Once your check boxes are selected, click the Map button to link the fields. The mapped fields now appear in the Mapped Fields section which lists all fields that have been mapped to each other.

You can perform the following actions in the Field Mapping page:

  • Use the circular arrow in the Mapped Fields section to undo a mapping.

  • Click the filter icon in the Unmapped Fields section to enter text for searching.

  • In the Unmapped Fields section, you can select multiple fields from the Parsed Fields column and map them to one field from the Insights Fields column. When you do this, a “Sort” icon appears in the Mapped Fields section. Use the Sort capability to select the order in which multiple fields are applied based on whether those fields contain a valid value or not. Higher in the order takes priority.

  • Select the Counter check box to count the number of times a field appears.

Date Format

Field Mapping: Format Date and Time

This is an optional configuration and can be left blank, if your log file is using a standard time as dictated by RFC 3164 or RFC 5424. Those headers are automatically parsed. If the timestamp cannot be parsed, use the Ruby strftime to provide a format string so that Security Director Insights can interpret the date and time in your log file as the event start time.

For more information on Ruby strftime format, see https://ruby-doc.org/core-2.3.0/Time.html#method-i-strftime.

Log Filtering

Log Filtering

You can create filters to notify Security Director Insights which events are malicious and which are not as you decide what logs are to be kept and which ones can be ignored. This removes logs that are “noisy” and not of particular interest and keep logs that are related to malicious events.

With these filters, you can select “exact match” filters or “contains” for the string you enter.

Click Add and configure filtering conditions as follows:

  • Select a log file field from the list.

  • Select a suitable condition from the list such as Matches, Contains, Does not Contain, and so on. If you select Matches, your provided string must match the selected field exactly. If you select Contains, your provided string must appear as a substring within the selected field.

  • In the edit field, enter a string to filter log files and click Add.

Click OK and your condition is added to the filter. You can add multiple filters. An “or” condition is applied to the list of filters, therefore the order of filters is not relevant.

Note: Select the check box and click Delete to remove a filter.

Conditions Assignment

Assign Conditions

You can assign different conditions to an event, based on filtering parameter you configure.

  • Event Severity—Assign conditions to define the severity of an event.

    Click Add and set conditions as follows:

    • Select a Severity level. The options are: Benign, Low, Medium, High, and Critical.

    • Select a field from the list to set the severity level for that field.

    • Select a condition. For example, If you select Matches, your string must match the selected field exactly. If you select Contains, your string must appear as a substring within the selected field.

    • In the edit field, enter a string to filter log files and click Add.

  • Progression—Assign conditions to define the progression of an event.

    Click Add and set conditions as follows:

    • Select a progression level. The options are: Phishing, Exploit, Download, Infection, and Execution.

    • Select a field from the list to set the progression level for that field.

    • Select a condition. For example, If you select Matches, your string must match the selected field exactly. If you select Contains, your string must appear as a substring within the selected field.

    • In the edit field, enter a string to filter log files and click Add.

  • Blocked—Assign conditions to define the event is blocked or not.

    Click Add and set conditions as follows:

    • Select a blocked level. The options are: True, False.

    • Select a field from the list to set the block level for that field.

    • Select a condition. For example, If you select Matches, your string must match the selected field exactly. If you select Contains, your string must appear as a substring within the selected field.

    • In the edit field, enter a string to filter log files and click Add.