Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?


Parsing Issues and Examples


When you create a log source extension, you might encounter some parsing issues. Use these XML examples to resolving specific parsing issues.

Converting a Protocol

The following example shows a typical protocol conversion that searches for TCP, UDP, ICMP, or GRE anywhere in the payload. The search pattern is surrounded by any word boundary, for example, tab, space, end of line. Also, the character case is ignored:

<pattern id="Protocol" case-insensitive="true" xmlns=""> <![CDATA[\b(TCP|UDP|ICMP|GRE)\b]]> </pattern> <matcher field="Protocol" order="1" pattern-id="Protocol" capture-group="1" />

Making a Single Substitution

The following example shows a substitution that parses the source IP address, and then overrides the result and sets the IP address to, ignoring the IP address in the payload.

This example assumes that the source IP address matches something similar to SrcAddress= followed by a comma:

<pattern id="SourceIp_AuthenOK" xmlns=""> <![CDATA[SrcAddress=(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}),]]> </pattern> <matcher field="SourceIp" order="1" pattern-id="SourceIp_AuthenOK" capture-group="" enable-substitutions="true"/>

Generating a Colon-separated MAC Address

JSA detects MAC addresses in a colon-separated form. Because all devices might not use this form, the following example shows how to correct that situation:

<pattern id="SourceMACWithDashes" xmlns=""> <![CDATA[SourceMAC=([0-9a-fA-F]{2})-([0-9a-fA-F]{2})-([0-9a-fA-F]{2})- ([0-9a-fA-F]{2})-([0-9a-fA-F]{2})-([0-9a-fA-F]{2})]]> </pattern> <matcher field="SourceMAC" order="1" pattern-id=" SourceMACWithDashes" capture-group="\1:\2:\3:\4:\5:\6" />

In the preceding example, SourceMAC=12-34-56-78-90-AB is converted to a MAC address of 12:34:56:78:90:AB.

If the dashes are removed from the pattern, the pattern converts a MAC address and has no separators. If spaces are inserted, the pattern converts a space-separated MAC address.

Combining IP Address and Port

Typically an IP address and port are combined into one field, which is separated by a colon.

The following example uses multiple capture groups with one pattern:

pattern id="SourceIPColonPort" xmlns=""> <! [CDATA[Source=(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}):([\d]{1,5})]]> </pattern> <matcher field="SourceIp" order="1" pattern-id="SourceIPColonPort" capture-group="1" /> <matcher field="SourcePort" order="1" pattern-id="SourceIPColonPort" capture-group="2" />

Modifying an Event Category

A device event category can be hardcoded, or the severity can be adjusted.

The following example adjusts the severity for a single event type:

<event-match-single event-name="TheEvent" device-event-category="Actual Category" severity="6" send-identity="UseDSMResults" />

Suppressing Identity Change Events

A DSM might unnecessarily send identity change events.

The following examples show how to suppress identity change events from being sent from a single event type and a group of events.

// Never send identity for the event with an EventName of Authen OK <event-match-single event-name="Authen OK" device-event-category="ACS" severity="6" send-identity="OverrideAndNeverSend" /> // Never send any identity for an event with an event name starting with 7, followed by one to five other digits: <pattern id="EventNameId" xmlns=""><![CDATA[(7\d{1,5})]]> </pattern> <event-match-multiple pattern-id="EventNameId" capture-group-index="1" device-event-category="Cisco Firewall" severity="7" send-identity="OverrideAndNeverSend"/>

Formatting Event Dates and Time Stamps

A log source extension can detect several different date and time stamp formats on events.

Because device manufacturers do not conform to a standard date and time stamp format, the ext-data optional parameter is included in the log source extension to allow the DeviceTime to be reformatted. The following example shows how an event can be reformatted to correct the date and time stamp formatting:

<device-extension> <pattern id="EventName1">(logger):</pattern> <pattern id="DeviceTime1">time=\[(\d{2}/\w{3}/\d{4}:\d{2}:\d{2}:\d{2})\]</pattern> <pattern id="Username">(TLSv1)</pattern> <match-group order="1" description="Full Test"> <matcher field="EventName" order="1" pattern-id="EventName1_Pattern" capture-group="1"/> <matcher field="DeviceTime" order="1" pattern-id="DeviceTime1_Pattern" capture-group="1" ext-data="dd/MMM/YYYY:hh:mm:ss"/> <matcher field="UserName" order="1" pattern-id="Username_Pattern" capture-group="1"/> </match-group> </device-extension>

Multiple Log Formats in a Single Log Source

Occasionally, multiple log formats are included in a single log source.

May 20 17:15:50 kernel: DROP IN=vlan2 OUT= MAC= SRC= DST= PROTO=UDP SPT=1900 DPT=1900 May 20 17:16:26 dropbear[22331]: password auth succeeded for 'root' from May 20 17:16:28 dropbear[22331]: exit after auth (root): Exited normally </br> May 20 17:16:14 dropbear[22331]: bad password attempt for 'root' from

For example, there are 2 log formats: one for firewall events, and one for authentication events. You must write multiple patterns for parsing the events. You can specify the order to be parsed. Typically, the more frequent events are parsed first, followed by the less frequent events. You can have as many patterns as required to parse all of the events. The order variable determines what order the patterns are matched in.

The following example shows multiple formats for the following fields EventName and UserName

Separate patterns are written to parse each unique log type. Both of the patterns are referenced when you assign the value to the normalized fields.

<pattern id="EventName-DDWRT-FW_Pattern" xmlns=""><![CDATA[kernel\:\s(.*?)\s]]></pattern> <pattern id="EventName-DDWRT-Auth_Pattern" xmlns=""><![CDATA[sdrophear\[\d{1,5}\]|:\s(.*?\s.*?)\s]]> </pattern> <pattern id="UserName_DDWRT-Auth1__Pattern" xmlns=""><![CDATA[\sfor\s\'(.*?)\'s]]></pattern> <pattern id="UserName_DDWRT-Auth2__Pattern" xmlns=""><![CDATA[\safter\sauth\s\((.*?)\)\:]]></pattern> <match-group order="1" description="DD-WRT Device Extensions xmlns=""> <matcher field="EventName" order="1" pattern-id="EventName-DDWRT-FW_Pattern" capture-group="1"/> <matcher field="EventName" order="2" pattern-id="EventName-DDWRT-Auth_Pattern" capture-group="1"/> <matcher field="UserName" order="1" pattern-id="UserName-DDWRT-Auth1_Pattern" capture-group="1"/> <matcher field="UserName" order="2" pattern-id="UserName-DDWRT-Auth2_Pattern" capture-group="1"/>

Parsing a CSV Log Format

A CSV-formatted log file can use a single parser that has multiple capture groups. It is not always necessary to create multiple Pattern IDs when you parse this log type.

The following log sample is used:

Event,User,Source IP,Source Port,Destination IP,Destination Port Failed Login,bjones,,1024,,22 Successful Login,nlabadie,,1743,,110 Privilege Escalation,bjones,,1028,,23
  1. Create a parser that matches all relevant values by using the previous patterns.
    .*?\,.*?\,\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3} \,\d{1,5}\,\d{1,3}\.\d{1,3} \.\d{1,3}\.\d{1,3}\,\d{1,5}
  2. Place the capture groups around each value:
    (.*?)\,(.*?)\,(\d{1,3}\.\d{1,3}\.\d{1,3}\. \d{1,3})\,(\d{1,5})\,(\d{1,3} \.\d{1,3}\.\d{1,3}\.\d{1,3})\,(\d{1,5})
  3. Map the field that each capture group is mapped to, incrementing the value as you move.
    1 = Event, 2 = User, 3 = Source IP, 4 = Source Port, 5 = Destination IP, 6 = Destination Port
  4. Include the values in the log source extension by mapping the capture group to the relevant event.

    The following code shows a partial example of mapping the capture group to the relevant event.

    <pattern id="CSV-Parser_Pattern" xmlns=""><![CDATA 9.*?)\,(.*?)\,(\d{1,3}\.\{1,3}\.{1,3}]]></pattern> <match-group order="1" description="Log Source Extension xmlns=""> <matcher field="EventName" order="1" pattern-id="CSV-Parser_Pattern" capture-group="1"/> <matcher field="SourceIP" order="1" pattern-id="CSV-Parser_Pattern" capture-group="3"/> <matcher field="SourcePort" order="1" pattern-id="CSV-Parser_Pattern" capture-group="4"/> <matcher field="DestinationIP" order="1" pattern-id="CSV-Parser_Pattern" capture-group="5"/> <matcher field="DestinationPort" order="1" pattern-id="CSV-Parser_Pattern" capture-group="6"/> <matcher field="UserName" order="1" pattern-id="CSV-Parser_Pattern" capture-group="2"/>
  5. Upload the log source extension.
  6. Map the events.