Property Configuration in the DSM Editor

Referencing Capture Strings by using Format String Fields

Use the Format String field on the Property Configuration tab to reference capture groups that you defined in the regex. Capture groups are referenced in their order of precedence.

A capture group is any regex that is enclosed within parenthesis. A capture group is referenced with an $n notation, where n is a group number that contains a regular expression (regex). You can define multiple capture groups.

For example, you have a payload with company and host name variables.

"company":"companyname", "hostname":"localhost.com" "company":"companyname", "hostname":"username.com"

You can customize the host name from the payload to display example.hostname.com by using capture groups:

1. In the regex field, enter the following regular expression:

   "company":"(.*?)".*"hostname":"(.*?)"

2. In the Format String field, enter the capture group $1.$2 where $1 is the value for the company variable and $2 is the value for the host name in the payload.

   The following output is given:

   companyname.localhost.com

   companyname.username.com

Regex for Well-structured Logs

Well-structured logs are a style of event formatting that is composed of a set of properties and are presented in the following way:

<name_of_property_1> <assignment_character> <value_of_property_1> <delimiter_character> <name_of_property_2> <assignment_character> <value_of_property_2> <delimiter_character> <name_of_property_3> <assignment_character> <value_of_property_3> <delimiter_character>...

Use the following general guidelines:

• The <assignment_character> either '=' or ':' or a multi-character sequence such as '->'.

• The <delimiter_character> either a white space character (space or tab) or a list delimiter, such as a comma or semi-colon.

• The <value_of_property> and sometimes <name_of_property> are encapsulated in quotation marks or other wrapping characters.

For example, consider a simple login event that is generated by a device or an application. The device might report on the account of a user who logged in, the time the login occurred, and the IP address of the computer from which the user logged in. A name/value pair-style event might look like this snippet:

<13>Sep 09 22:40:40 192.0.2.12 action=login accountname=JohnDoe clientIP= 192.0.2.24 timestamp=01/09/2016 22:40:39 UTC

Note:

The string "<13>Sep 09 22:40:40 192.0.2.12" is a syslog header. The string is not part of the event body.

The following table shows how the properties of the well-structured log example above, can be captured:

Table 1: Regex for Capturing Properties Of a Well-structured Log

Property	Regex
action	action=(.*?)\t
accountname	accountname=(.*?)\t
clientIP	clientIP=(.*?)\t
timestamp	timestamp=(.*?)\t

The patterns that are enclosed within the brackets denote the capture group. Each regex in the table captures everything after the equal sign (=) and before the next tab character.

Regex for Natural Language Logs

Natural language logs are presented in a sentence-like form and each event type might look different.

For example, a simple login event can be presented in the following form:

<13>Sep 09 22:40:40 192.0.2.12 Account JohnDoe initiated a login action from 192.0.2.24 at 01/09/2016 22:40:39 UTC

The following table shows how the properties of the natural language log in the example above, can be captured:

Table 2: Regex for Capturing Properties Of a Natural Language Log

Property	Regex
action	initiated a (.*?) action
accountname	Account (.*?) initiated
clientIP	from (.*?) at
timestamp	at (.*?)

Note:

Writing regex for natural language logs requires you to look at the static information that surrounds the value you want to capture before you create the capture group.

Expressions in JSON Format for Structured Data

Structured data in JSON format contains one or more properties, which are represented as a key-value pair.

You can extract properties from an event data presented in JSON format by writing a JSON expression that matches the property. The JSON expression must be a path in the format of /"<name of top-level field>".

For example, you have an event data formatted in JSON:

or an event that has a nested JSON format, such as:

To extract properties from event data, choose one of the following methods:

• To extract the 'user' property for event data that is formatted in JSON, type the expression /"user" in the Expression field.

• To extract the 'last_name' of the user for an event that has a nested JSON format, type the expression /"user"/"last_name" in the Expression field.

JSON Keypath Expressions

To uniquely identify the fields that you want to extract from a JSON object, your JSON expression must follow specific JSON keypath conventions.

Use the following guidelines for your JSON keypath expressions:

• A forward slash (/) must be at the start of all JSON keypaths. All paths must start at the beginning of the root JSON object. Subsequent slashes in the keypath indicate access to fields that are nested in the JSON object.

• Field names must be enclosed in double quotation marks.

  A valid path might look like the following example:

• Square brackets indicate the handling of JSON arrays.

  If you do not supply an index in the square brackets, the entire body of the array is extracted. If you supply an index in the square bracket, that index in the array is extracted or nested. Arrays begin at a zero index, where 0 is the first index in the array, 1 is the second index in the array, and so on.

  In the following keypath example, the JSON parser looks into the second index of the "object" JSON array, and then within that array index, looks for a field called "desiredPropertyName".

• Within log source extensions, you can supply and combine together multiple JSON keypaths to give a single result; this convention excludes custom properties. You can also choose to include literal text. Each of the JSON keypaths must be enclosed in curly braces.

  Consider the following example:

  You get a parsed value from the first JSON keypath, a literal text space, and then a parsed value from the second JSON keypath.

  Example: The following two examples show how to extract data from a JSON object:

  The following table shows the values that are extractable from the keypaths in that sample object:

• Simple case of a JSON object:

  The following table shows the values that are extractable from the keypaths in that sample object:

  Table 3: Keypaths from the Simple JSON Object

  Keypath	Description	Value
  /[]	Extracts the entire JSON array from the root of the JSON object	[{"name":"object1","field1":"value1"}, {"name":"object2","field2":"value2"}, {"name":"object3","field3":"value3"}]
  /[1]/"name"	Extracts the value for the attribute called "name" from the JSON object at index 1 in the root JSON array.	object2

• Complex case of a JSON object:

  <13>May 22 10:15:41 log.test.com {"module":"CPHalo","version":"1.0","user_name":"user123", "event_type":"File integrity scan request created", "event_category":"File Integrity Scanning Management","srcName":"domain-lab-123", "timestamp":"2018-12-02T15:36:17.486","user": {"email":"user123@example.com","first_name":"fname", "last_name":"lname","alias":["alias name","alias1","name"]},"client_ip":"10.12.12.12", "server_id":"12317412471421274","server_reported_fqdn":"None","actor_country":"USA", "server_group_name":"Example Server","server_platform":"Linux", "message":"A file integrity monitoring scan was requested for Linux server domain-lab-123 (10.13.13.13) by Halo user user123@example.com from IP address 10.12.12.12 (USA).", "type":"fim_scan_request_created","id":"c2e8bf72- b74f-11e2-9055-870a490fcfb6"}

  The following table shows the values that are extractable from the keypaths in that sample object:

  Table 4: Keypaths from the Complex JSON Object

  Keypath	Description	Value
  /"user_name"	Extracts value of the "user_name" attribute from the root of the JSON object.	user123
  /"user"/"alias"[]	Extracts the entire JSON array called "alias" that is nested under the "user" JSON object.	["alias name","alias1","name"]
  /"user"/"alias"[0]	Extracts the value at index 0 within the "alias" JSON array that is nested under the "user" JSON Object.	alias name
  /"user"/'first_name"	Extracts the value of the property called "first_name" that is nested under the "user" JSON Object.	fname
  {/"user"/"first_name"}.{/"user"/ "last_name"}	Extracts the value of the property called "first_name" that is nested under the "user" JSON object, then inserts a literal '.' character, and then extracts the value of the property called "second_name" that is nested under the "user" JSON object. Pertains only to log source extensions and non-custom properties within the DSM Editor. This operation is not possible in custom properties.	fname.lname
  {/"user"/"alias"[1]}@{/"client_ip"}	Extracts the value at index 1 of the "alias" JSON array that is nested under the "user' JSON object, inserts a literal '@' character, and then extracts the value of the property called "client_ip" under the root JSON object. Pertains only to log source extensions and non-custom properties within the DSM Editor. This operation is not possible in custom properties.	alias1@10.12.12.12

Expressions in LEEF Format for Structured Data

Structured data in LEEF format contains one or more properties, which are represented as key-value pairs.

You can extract properties from an event that is presented in LEEF format by writing a LEEF expression that matches the property. Valid LEEF expressions are in the form of either a single key reference, or a special LEEF header field reference.

For example, you have an event that is formatted in LEEF V1.0, such as:

or an event that is formatted in LEEF V2.0 with the caret (^) separator character, such as:

You can extract a property or a header key property from the event by choosing one of the following methods:

1. To extract the 'usrName' property, enter usrName in the LEEF Key field.

   The possible keys that can be extracted are:

   • devTimeFormat

   • devTime

   • usrName

   • name

   • authType

   • src

2. To extract a header key property, type the key in the following format in the LEEF Key field:

   $eventid$

   The LEEF header values can be extracted by using the following expressions:

   • $leefversion$

   • $vendor$

   • $product$

   • $version$

   • $eventid$

Expressions in CEF Format for Structured Data

Structured data in CEF format contains one or more properties, which are represented as key-value pairs.

You can extract properties from an event that is presented in CEF format by writing a CEF expression that matches the property. Valid CEF expressions are in the form of either a single key reference, or a special CEF header field reference.

For example, you have an event that is formatted in CEF:

You can extract a property or a header key property from the event by choosing one of the following methods:

1. To extract the 'cs1' property, type cs1 in the CEF Key field.

   The possible keys that can be extracted are:

   • start

   • duser

   • cs1

   • cs1Label

   • cs2

   • cs2Label

   • src

2. To extract a header key property, type the key in the following format in the CEF Key field:

   $id$

   The CEF header values can be extracted by using the following expressions:

   • $cefversion$

   • $vendor$

   • $product$

   • $version$

   • $id$

   • $name$

   • $severity$

Expressions in Name Value Pair Format for Structured Data

Structured data in Name Value Pair format contains one or more properties, which are represented as key-value pairs.

You can extract properties from an event that is in Name Value Pair format by writing an expression that matches the property. Valid Name Value Pair expressions are in the form of a single key reference.

The following example shows an event that is in Name Value Pair format:

To extract the Username property, type Username in the Expression field.

In the Value Delimiter field, enter the key-value delimiter that is specific for your payload. In this example, the keyvalue delimiter is an equal sign (=).

In the Delimiter field, enter the delimiter between key-value pairs that is specific for your payload. In this example, the delimiter between key-value pairs is a semicolon (;).

Matches in the payload are highlighted in the event data in the Workspace of the DSM Editor.

Expressions in Generic List Format for Structured Data

Structured data in Generic List format contains one or more properties, which are represented as list items.

You can extract properties from an event that is in Generic List format by writing an expression that matches the property. Valid Generic List expressions are in the form of a $<number> notation. For example, $0 represents the first property in the list, $1 is the second property, and so on.

The following example shows an event that is in Generic List format:

To extract the first property in the list, type $0 in the Expression field.

In the Delimiter field, enter the delimiter between list items that is specific for your payload. In this example, the delimiter between list items is a semicolon (;).

Matches in the payload are highlighted in the event data in the Workspace of the DSM Editor.

Expressions in XML Format for structured data

Structured data in XML format contains one or more properties, which are represented as key-value pairs.

You can extract properties from an event that is in XML format by writing an expression that matches the property. Valid XML expressions are in the form of a single key reference.

Enter the path to the XML field that you want to use to populate the property's value. An XML key path must begin with a forward slash (/) to indicate the root of the XML object, and be followed by one or more XML field names within double quotation marks.

The following example shows an event that is in XML format:

To capture the value nested in the top-level OSName object, type /"EPOEvent"/"MachineInfo"/"OSName" in the Expression field.

To capture the attribute value, use a period (.) after the key path. For example, to capture some attribute value, type /"EPOEvent"/"MachineInfo"/"IPAddress".someattrib in the Expression field.

To combine multiple fields together with multiple paths, use set brackets to enclose each. For example,

{/"EPOEvent"/"MachineInfo"/"OSName"} {/"EPOEvent"/"MachineInfo"/"MachineName"[1]}

To capture the value that is nested within multiple tags with the same name, use [0], [1], and so on, after the key path. For example, to capture VALUE23, type, /"EPOEvent"/"MachineInfo"/"MachineName"[1] in the Expression field.

Matches in the payload are highlighted in the event data in the Workspace of the DSM Editor.