Parsing Issues and Examples
When you create a log source extension, you might encounter some parsing issues. Use these XML examples to resolving specific parsing issues.
Converting a Protocol
The following example shows a typical protocol conversion that searches for TCP, UDP, ICMP, or GRE anywhere in the payload. The search pattern is surrounded by any word boundary, for example, tab, space, end of line. Also, the character case is ignored:
<pattern id="Protocol" case-insensitive="true" xmlns="">
<![CDATA[\b(TCP|UDP|ICMP|GRE)\b]]> </pattern>
<matcher field="Protocol" order="1" pattern-id="Protocol" capture-group="1"
/>
Making a Single Substitution
The following example shows a substitution that parses the source IP address, and then overrides the result and sets the IP address to 192.0.2.1, ignoring the IP address in the payload.
This example assumes that the source IP address matches something
similar to SrcAddress=203.0.113.1
followed
by a comma:
<pattern id="SourceIp_AuthenOK" xmlns=""> <![CDATA[SrcAddress=(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}),]]> </pattern>
<matcher field="SourceIp" order="1" pattern-id="SourceIp_AuthenOK"
capture-group="192.0.2.1" enable-substitutions="true"/>
Generating a Colon-separated MAC Address
JSA detects MAC addresses in a colon-separated form. Because all devices might not use this form, the following example shows how to correct that situation:
<pattern id="SourceMACWithDashes" xmlns=""> <![CDATA[SourceMAC=([0-9a-fA-F]{2})-([0-9a-fA-F]{2})-([0-9a-fA-F]{2})-
([0-9a-fA-F]{2})-([0-9a-fA-F]{2})-([0-9a-fA-F]{2})]]> </pattern>
<matcher field="SourceMAC" order="1" pattern-id=" SourceMACWithDashes"
capture-group="\1:\2:\3:\4:\5:\6" />
In the preceding example, SourceMAC=12-34-1a-2b-3c-4d
is converted to a MAC address of 12:34:1a:2b:3c:4d
.
If the dashes are removed from the pattern, the pattern converts a MAC address and has no separators. If spaces are inserted, the pattern converts a space-separated MAC address.
Combining IP Address and Port
Typically an IP address and port are combined into one field, which is separated by a colon.
The following example uses multiple capture groups with one pattern:
pattern id="SourceIPColonPort" xmlns=""> <! [CDATA[Source=(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}):([\d]{1,5})]]>
</pattern> <matcher field="SourceIp" order="1" pattern-id="SourceIPColonPort" capture-group="1" /> <matcher field="SourcePort" order="1"
pattern-id="SourceIPColonPort" capture-group="2" />
Modifying an Event Category
A device event category can be hardcoded, or the severity can be adjusted.
The following example adjusts the severity for a single event type:
<event-match-single event-name="TheEvent"
device-event-category="Actual Category" severity="6" send-identity="UseDSMResults"
/>
Suppressing Identity Change Events
A DSM might unnecessarily send identity change events.
The following examples show how to suppress identity change events from being sent from a single event type and a group of events.
// Never send identity for the event with an EventName of
Authen OK <event-match-single event-name="Authen OK" device-event-category="ACS"
severity="6" send-identity="OverrideAndNeverSend" /> // Never send
any identity for an event with an event name starting with 7, followed
by one to five other digits: <pattern id="EventNameId" xmlns=""><![CDATA[(7\d{1,5})]]>
</pattern> <event-match-multiple pattern-id="EventNameId" capture-group-index="1"
device-event-category="Cisco Firewall" severity="7" send-identity="OverrideAndNeverSend"/>
Formatting Event Dates and Time Stamps
A log source extension can detect several different date and time stamp formats on events.
Because device manufacturers do not conform to a standard date and time stamp format, the ext-data optional parameter is included in the log source extension to allow the DeviceTime to be reformatted. The following example shows how an event can be reformatted to correct the date and time stamp formatting:
<device-extension> <pattern id="EventName1">(logger):</pattern>
<pattern id="DeviceTime1">time=\[(\d{2}/\w{3}/\d{4}:\d{2}:\d{2}:\d{2})\]</pattern>
<pattern id="Username">(TLSv1)</pattern> <match-group
order="1" description="Full Test"> <matcher field="EventName"
order="1" pattern-id="EventName1_Pattern" capture-group="1"/> <matcher field="DeviceTime" order="1" pattern-id="DeviceTime1_Pattern"
capture-group="1" ext-data="dd/MMM/YYYY:hh:mm:ss"/>
<matcher field="UserName" order="1" pattern-id="Username_Pattern"
capture-group="1"/> </match-group> </device-extension>
Multiple Log Formats in a Single Log Source
Occasionally, multiple log formats are included in a single log source.
May 20 17:15:50 kernel: DROP IN=vlan2 OUT= MAC= SRC=67.149.62.133
DST=239.255.255.250 PROTO=UDP SPT=1900 DPT=1900 May 20 17:16:26 dropbear[22331]:
password auth succeeded for 'root' from 192.168.50.80:3364 May 20
17:16:28 dropbear[22331]: exit after auth (root): Exited normally
</br> May 20 17:16:14 dropbear[22331]: bad password attempt for
'root' from 192.168.50.80:3364
For example, there are 2 log formats: one for firewall events, and one for authentication events. You must write multiple patterns for parsing the events. You can specify the order to be parsed. Typically, the more frequent events are parsed first, followed by the less frequent events. You can have as many patterns as required to parse all of the events. The order variable determines what order the patterns are matched in.
The following example shows multiple formats for the following fields EventName and UserName
Separate patterns are written to parse each unique log type. Both of the patterns are referenced when you assign the value to the normalized fields.
<pattern id="EventName-DDWRT-FW_Pattern" xmlns=""><![CDATA[kernel\:\s(.*?)\s]]></pattern>
<pattern id="EventName-DDWRT-Auth_Pattern" xmlns=""><![CDATA[sdrophear\[\d{1,5}\]|:\s(.*?\s.*?)\s]]>
</pattern> <pattern id="UserName_DDWRT-Auth1__Pattern" xmlns=""><![CDATA[\sfor\s\'(.*?)\'s]]></pattern>
<pattern id="UserName_DDWRT-Auth2__Pattern" xmlns=""><![CDATA[\safter\sauth\s\((.*?)\)\:]]></pattern>
<match-group order="1" description="DD-WRT Device Extensions xmlns="">
<matcher field="EventName" order="1" pattern-id="EventName-DDWRT-FW_Pattern"
capture-group="1"/> <matcher field="EventName" order="2" pattern-id="EventName-DDWRT-Auth_Pattern"
capture-group="1"/> <matcher field="UserName" order="1"
pattern-id="UserName-DDWRT-Auth1_Pattern" capture-group="1"/>
<matcher field="UserName" order="2" pattern-id="UserName-DDWRT-Auth2_Pattern"
capture-group="1"/>
Parsing a CSV Log Format
A CSV-formatted log file can use a single parser that has multiple capture groups. It is not always necessary to create multiple Pattern IDs when you parse this log type.
The following log sample is used:
Event,User,Source IP,Source Port,Destination IP,Destination
Port Failed Login,bjones,192.168.50.100,1024,10.100.24.25,22 Successful
Login,nlabadie,192.168.64.76,1743,10.100.24.25, 110 Privilege Escalation,bjones,192.168.50.100,1028,10.100.1.100,
23
- Create a parser that matches all relevant values by using
the previous patterns.
.*?\,.*?\,\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3} \,\d{1,5}\,\d{1,3}\.\d{1,3} \.\d{1,3}\.\d{1,3}\,\d{1,5}
- Place the capture groups around each value:
(.*?)\,(.*?)\,(\d{1,3}\.\d{1,3}\.\d{1,3}\. \d{1,3})\,(\d{1,5})\,(\d{1,3} \.\d{1,3}\.\d{1,3}\.\d{1,3})\,(\d{1,5})
- Map the field that each capture group is mapped to, incrementing
the value as you move.
1 = Event, 2 = User, 3 = Source IP, 4 = Source Port, 5 = Destination IP, 6 = Destination Port
- Include the values in the log source extension by mapping
the capture group to the relevant event.
The following code shows a partial example of mapping the capture group to the relevant event.
<pattern id="CSV-Parser_Pattern" xmlns=""><![CDATA 9.*?)\,(.*?)\,(\d{1,3}\.\{1,3}\.{1,3}]]></pattern> <match-group order="1" description="Log Source Extension xmlns=""> <matcher field="EventName" order="1" pattern-id="CSV-Parser_Pattern" capture-group="1"/> <matcher field="SourceIP" order="1" pattern-id="CSV-Parser_Pattern" capture-group="3"/> <matcher field="SourcePort" order="1" pattern-id="CSV-Parser_Pattern" capture-group="4"/> <matcher field="DestinationIP" order="1" pattern-id="CSV-Parser_Pattern" capture-group="5"/> <matcher field="DestinationPort" order="1" pattern-id="CSV-Parser_Pattern" capture-group="6"/> <matcher field="UserName" order="1" pattern-id="CSV-Parser_Pattern" capture-group="2"/>
- Upload the log source extension.
- Map the events.