Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

 

Property Configuration in the DSM Editor

 

Configure properties in the DSM Editor to change the behavior of an overridden system property or the custom property of a DSM.

When you override the behavior of a system property, you must provide a valid expression on the Property Configuration tab. The Format String field is a combination of regex capture groups and literal characters. The string is used to populate system properties by one or more values that are captured from events, and with more formatting characters or injected information. For example, you might want to parse an IP address and a port to combine them both into a string. If your regular expression (regex) has two capture groups, you can combine them by using this format string: $1:$2.

Note

The DSM Editor allows capture group references of 1 through 9 in any given match. If you reference any capture group above 9, the log source extension might not work correctly.

You must configure each custom property that you create. You must provide a valid expression and capture group for a custom property on the Property Configuration tab. You can also define selectivity and enable or disable your expression.

Writing Format String to Use Capture Strings

Use the Format String field on the Property Configuration tab to reference capture groups that you defined in the regex. Capture groups are referenced in their order of precedence.

A capture group is any regex that is enclosed within parenthesis. A capture group is referenced with an $n notation, where n is a group number that contains a regular expression (regex). You can define multiple capture groups.

For example, you have a payload with company and host name variables.

"company":"ibm", "hostname":"localhost.com"

"company":"ibm", "hostname":"johndoe.com"

You can customize the host name from the payload to display ibm.hostname.com by using capture groups:

  1. In the regex field, enter the following regular expression:

    "company":"(.*?)".*"hostname":"(.*?)"

  2. In the Format String field, enter the capture group $1.$2 where $1 is the value for the company variable (in this case ibm) and $2 is the value for the host name in the payload.

    The following output is given:

    ibm.localhost.com

    ibm.johndoe.com

Writing Regex for Well-structured Logs

Well-structured logs are a style of event formatting that is composed of a set of properties and are presented in this way:

<name_of_property_1> <assignment_character>

<value_of_property_1> <delimiter_character>

<name_of_property_2> <assignment_character>

<value_of_property_2> <delimiter_character>

<name_of_property_3> <assignment_character>

<value_of_property_3> <delimiter_character>...

Use the following general guidelines:

  • The <assignment_character> either '=' or ':' or a multi-character sequence such as '->'.

  • The <delimiter_character> either a white space character (space or tab) or a list delimiter, such as a comma or semi-colon.

  • The <value_of_property> and sometimes <name_of_property> are encapsulated in quotation marks or other wrapping characters.

For example, consider a simple login event that is generated by a device or an application. The device might report on the account of a user who logged in, the time the login occurred, and the IP address of the computer from which the user logged in. A name/value pair-style event might look like this snippet:

<13>Sep 09 22:40:40 192.0.2.12 action=login accountname=JohnDoe clientIP= 192.0.2.24 timestamp=01/09/2016 22:40:39 UTC

Note

The string "<13>Sep 09 22:40:40 192.0.2.12" is a syslog header. The string is not part of the event body.

The following table shows how the properties of the well-structured log example above, can be captured:

Table 1: Regex for Capturing Properties Of a Well-structured Log

Property

Regex

action

action=(.*?)\t

accountname

accountname=(.*?)\t

clientIP

clientIP=(.*?)\t

timestamp

timestamp=(.*?)\t

The patterns that are enclosed within the brackets denote the capture group. Each regex in the table captures everything after the equal sign (=) and before the next tab character.

Writing Regex for Natural Language Logs

Natural language logs are presented in a sentence-like form and each event type might look different.

For example, a simple login event can be presented in the following form:

<13>Sep 09 22:40:40 192.0.2.12 Account JohnDoe initiated a login action from 192.0.2.24 at 01/09/2016 22:40:39 UTC

The following table shows how the properties of the natural language log in the example above, can be captured:

Table 2: Regex for Capturing Properties Of a Natural Language Log

Property

Regex

action

initiated a (.*?) action

accountname

Account (.*?) initiated

clientIP

from (.*?) at

timestamp

at (.*?)

Note

Writing regex for natural language logs requires you to look at the static information that surrounds the value you want to capture before you create the capture group.

Writing an Expression for Structured Data in JSON Format

Structured data in JSON format contains one or more properties, which are represented as a key-value pair.

You can extract properties from an event data presented in JSON format by writing a JSON expression that matches the property. The JSON expression must be a path in the format of /"<name of top-level field>".

For example, you have an event data formatted in JSON:

To extract the 'user' property, type the expression /"user" in the Expression field.

However, for an event data with a nested JSON format such as the following example:

You can extract the 'last_name' of the user by typing the expression /"user"/"last_name" in the Expression field.

Writing a JSON Keypath Expression

To uniquely identify the fields that you want to extract from a JSON object, your JSON expression must follow specific JSON keypath conventions.

Use the following guidelines for your JSON keypath expressions:

  • A forward slash (/) must be at the start of all JSON keypaths. All paths must start at the beginning of the root JSON object. Subsequent slashes in the keypath indicate access to fields that are nested in the JSON object.

  • Field names must be enclosed in double quotation marks.

    A valid path might look like the following example:

  • Square brackets indicate the handling of JSON arrays.

    If you do not supply an index in the square brackets, the entire body of the array is extracted. If you supply an index in the square bracket, that index in the array is extracted or nested. Arrays begin at a zero index, where 0 is the first index in the array, 1 is the second index in the array, and so on.

    In the following keypath example, the JSON parser looks into the second index of the "object" JSON array, and then within that array index, looks for a field called "desiredPropertyName".

  • Within log source extensions, you can supply and combine together multiple JSON keypaths to give a single result; this convention excludes custom properties. You can also choose to include literal text. Each of the JSON keypaths must be enclosed in curly braces.

    Consider the following example:

    You get a parsed value from the first JSON keypath, a literal text space, and then a parsed value from the second JSON keypath.

    Example: The following two examples show how to extract data from a JSON object:

    The following table shows the values that are extractable from the keypaths in that sample object:

  • Simple case of a JSON object:

    The following table shows the values that are extractable from the keypaths in that sample object:

    Table 3: Keypaths from the Simple JSON Object

    Keypath

    Description

    Value

    /[]

    Extracts the entire JSON array from the root of the JSON object

    [{"name":"object1","field1":"value1"}, {"name":"object2","field2":"value2"}, {"name":"object3","field3":"value3"}]

    /[1]/"name"

    Extracts the value for the attribute called "name" from the JSON object at index 1 in the root JSON array.

    object2

  • Complex case of a JSON object:

    <13>May 22 10:15:41 log.test.com {"module":"CPHalo","version":"1.0","user_name":"user123", "event_type":"File integrity scan request created", "event_category":"File Integrity Scanning Management","srcName":"domain-lab-123", "timestamp":"2018-12-02T15:36:17.486","user": {"email":"user123@example.com","first_name":"fname", "last_name":"lname","alias":["alias name","alias1","name"]},"client_ip":"12.12.12.12", "server_id":"12317412471421274","server_reported_fqdn":"None","actor_country":"USA", "server_group_name":"Example Server","server_platform":"Linux", "message":"A file integrity monitoring scan was requested for Linux server domain-lab-123 (13.13.13.13) by Halo user user123@example.com from IP address 12.12.12.12 (USA).", "type":"fim_scan_request_created","id":"c2e8bf72- b74f-11e2-9055-870a490fcfb6"}

    The following table shows the values that are extractable from the keypaths in that sample object:

    Table 4: Keypaths from the Complex JSON Object

    Keypath

    Description

    Value

    /"user_name"

    Extracts value of the "user_name" attribute from the root of the JSON object.

    user123

    /"user"/"alias"[]

    Extracts the entire JSON array called "alias" that is nested under the "user" JSON object.

    ["alias name","alias1","name"]

    /"user"/"alias"[0]

    Extracts the value at index 0 within the "alias" JSON array that is nested under the "user" JSON Object.

    alias name

    /"user"/'first_name"

    Extracts the value of the property called "first_name" that is nested under the "user" JSON Object.

    fname

    {/"user"/"first_name"}.{/"user"/ "last_name"}

    Extracts the value of the property called "first_name" that is nested under the "user" JSON object, then inserts a literal '.' character, and then extracts the value of the property called "second_name" that is nested under the "user" JSON object.

    Pertains only to log source extensions and non-custom properties within the DSM Editor. This operation is not possible in custom properties.

    fname.lname

    {/"user"/"alias"[1]}@{/"client_ip"}

    Extracts the value at index 1 of the "alias" JSON array that is nested under the "user' JSON object, inserts a literal '@' character, and then extracts the value of the property called "client_ip" under the root JSON object.

    Pertains only to log source extensions and non-custom properties within the DSM Editor. This operation is not possible in custom properties.

    alias1@12.12.12.12

Writing an Expression for Structured Data in LEEF Format

Structured data in LEEF format contains one or more properties, which are represented as key-value pairs.

You can extract properties from an event that is presented in LEEF format by writing a LEEF expression that matches the property. Valid LEEF expressions are in the form of either a single key reference, or a special LEEF header field reference.

For example, you have an event that is formatted in LEEF V1.0, such as:

or an event that is formatted in LEEF V2.0 with the caret (^) separator character, such as:

To extract the 'usrName' property, type usrName in the LEEF Key field.

The possible keys that can be extracted in these examples are:

  • devTimeFormat

  • devTime

  • usrName

  • name

  • authType

  • src

To extract a header key property, type the key in the following format in the LEEF Key field:

The LEEF header values can be extracted by using the following expressions:

  • $leefversion$

  • $vendor$

  • $product$

  • $version$

  • $eventid$

Writing an Expression for Structured Data in CEF Format

Structured data in CEF format contains one or more properties, which are represented as key-value pairs.

You can extract properties from an event that is presented in CEF format by writing a CEF expression that matches the property. Valid CEF expressions are in the form of either a single key reference, or a special CEF header field reference.

For example, you have an event that is formatted in CEF:

To extract the 'cs1' property, type cs1 in the CEF Key field.

  • The possible keys that can be extracted in the example are:

    • start

    • duser

    • cs1

    • cs1Label

    • cs2

    • cs2Label

    • src

    To extract a header key property, type the key in the following format in the CEF Key field:

    The CEF header values can be extracted by using the following expressions:

    • $cefversion$

    • $vendor$

    • $product$

    • $version$

    • $id$

    • $name$

    • $severity$

Writing an Expression for Structured Data in Name Value Pair Format

Structured data in Name Value Pair format contains one or more properties, which are represented as key-value pairs.

You can extract properties from an event that is in Name Value Pair format by writing an expression that matches the property. Valid Name Value Pair expressions are in the form of a single key reference.

The following example shows an event that is in Name Value Pair format:

To extract the Username property, type Username in the Expression field.

In the Value Delimiter field, enter the key-value delimiter that is specific for your payload. In this example, the keyvalue delimiter is an equal sign (=).

In the Delimiter field, enter the delimiter between key-value pairs that is specific for your payload. In this example, the delimiter between key-value pairs is a semicolon (;).

Matches in the payload are highlighted in the event data in the Workspace of the DSM Editor.

Writing an Expression for Structured Data in Generic List Format

Structured data in Generic List format contains one or more properties, which are represented as list items.

You can extract properties from an event that is in Generic List format by writing an expression that matches the property. Valid Generic List expressions are in the form of a $<number> notation. For example, $0 represents the first property in the list, $1 is the second property, and so on.

The following example shows an event that is in Generic List format:

To extract the first property in the list, type $0 in the Expression field.

In the Delimiter field, enter the delimiter between list items that is specific for your payload. In this example, the delimiter between list items is a semicolon (;).

Matches in the payload are highlighted in the event data in the Workspace of the DSM Editor.