Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

 

Property Configuration in the DSM Editor

 

Configure properties in the DSM Editor to change the behavior of an overridden system property or the custom property of a DSM.

When you override the behavior of a system property, you must provide a valid regex and format string on the Property Configuration tab. The Format String field is a combination of regex capture groups and literal characters. The string is used to populate system properties by one or more values that are captured from events, and with more formatting characters or injected information. For example, you might want to parse an IP address and a port to combine them both into a string. If your regular expression (regex) has two capture groups, you can combine them by using this format string: $1:$2.

Note

The DSM Editor allows capture group references of 1 through 9 in any given match. If you reference any capture group above 9, the log source extension might not work correctly.

You must configure each custom property that you create. You must provide a valid regex and capture group for a custom property on the Property Configuration tab. You can also define selectivity and enable or disable your expression.

How to Write a Format String in the DSM Editor

Use the Format String field on the Property Configuration tab to reference capture groups that you defined in the regex. Capture groups are referenced in their order of precedence.

A capture group is any regex that is enclosed within parenthesis. A capture group is referenced with an $n notation, where n is a group number that contains a regular expression (regex). You can define multiple capture groups.

For example, you have a payload with company and host name variables.

"company":"ibm", "hostname":"localhost.com"

"company":"ibm", "hostname":"johndoe.com"

You can customize the host name from the payload to display ibm.hostname.com by using capture groups:

  1. In the regex field, enter the following regular expression:

    "company":"(.*?)".*"hostname":"(.*?)"

  2. In the Format String field, enter the capture group $1.$2 where $1 is the value for the company variable (in this case ibm) and $2 is the value for the host name in the payload.

    The following output is given:

    ibm.localhost.com

    ibm.johndoe.com

How to Write Regex for Well-structured Logs

Well-structured logs are a style of event formatting that is composed of a set of properties and are presented in this way:

<name_of_property_1> <assignment_character>

<value_of_property_1> <delimiter_character>

<name_of_property_2> <assignment_character>

<value_of_property_2> <delimiter_character>

<name_of_property_3> <assignment_character>

<value_of_property_3> <delimiter_character>...

Use the following general guidelines:

  • The <assignment_character> either '=' or ':' or a multi-character sequence such as '->'.

  • The <delimiter_character> either a white space character (space or tab) or a list delimiter, such as a comma or semi-colon.

  • The <value_of_property> and sometimes <name_of_property> are encapsulated in quotation marks or other wrapping characters.

For example, consider a simple login event that is generated by a device or an application. The device might report on the account of a user who logged in, the time the login occurred, and the IP address of the computer from which the user logged in. A name/value pair-style event might look like this snippet:

<13>Sep 09 22:40:40 9.2.45.12 action=login accountname=JohnDoe clientIP=9.21.23.24 timestamp=01/09/2016 22:40:39 UTC

Note

The string "<13>Sep 09 22:40:40 9.2.45.12" is a syslog header. The string is not part of the event body.

The following table shows how the properties of the well-structured log example above, can be captured:

Table 1: Regex for Capturing Properties Of a Well-structured Log

Property

Regex

action

action=(.*?)\t

accountname

accountname=(.*?)\t

clientIP

clientIP=(.*?)\t

timestamp

timestamp=(.*?)\t

The patterns that are enclosed within the brackets denote the capture group. Each regex in the table captures everything after the equal sign (=) and before the next tab character.

How to Write Regex for Natural Language Logs

Natural language logs are presented in a sentence-like form and each event type might look different.

For example, a simple login event can be presented in the following form:

<13>Sep 09 22:40:40 9.2.45.12 Account JohnDoe initiated a login action from 9.21.23.24 at 01/09/2016 22:40:39 UTC

The following table shows how the properties of the natural language log in the example above, can be captured:

Table 2: Regex for Capturing Properties Of a Natural Language Log

Property

Regex

action

initiated a (.*?) action

accountname

Account (.*?) initiated

clientIP

from (.*?) at

timestamp

at (.*?)

Note

Writing regex for natural language logs requires you to look at the static information that surrounds the value you want to capture before you create the capture group.

How to Write an Expression for Structured Data in JSON Format

Structured data in JSON format contains one or more properties, which are represented as a key-value pair.

You can extract properties from an event data presented in JSON format by writing a JSON expression that matches the property. The JSON expression must be a path in the format of /"<name of top-level field>".

For example, you have an event data formatted in JSON:

To extract the 'user' property, type the expression /"user" in the Expression field.

However, for an event data with a nested JSON format such as the following example:

You can extract the 'last_name' of the user by typing the expression /"user"/"last_name" in the Expression field.