Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

Creating a Log Source Extensions Document to get data into JSA

 

You create log source extensions (LSX) when log sources don't have a supported DSM, or to repair an event that has missing or incorrect information, or to parse an event when the associated DSM fails to produce a result.

When to create a Log Source Extension

For log sources that don't have an official DSM, use a Universal DSM (uDSM) to integrate log sources. A log source extension (also known as a device extension) is then applied to the uDSM to provide the logic for parsing the logs. The LSX is based on Java regular expressions and can be used against any protocol type, such as syslog, JDBC, and Log File. Values can be extracted from the logs and mapped to all common fields within JSA.

When you use log source extensions to repair missing or incorrect content, any new events that are produced by the log source extensions are associated to the log source that failed to parse the original payload. Creating an extension prevents unknown or uncategorized events from being stored as unknown in JSA.

Using the DSM Editor to quickly create a Log Source Extension

For JSA 2014.8 and later, you can use the DSM Editor to create log source extensions. The DSM Editor provides real-time feedback so that you know whether the log source extension that you are creating has problems. You use the DSM Editor to extract fields, define custom properties, categorize events, and define new QID definitions. You can use the DSM Editor to define your own Log Source Type, which eliminates the need to use a Universal DSM. For more information about the DSM Editor, see the Juniper Secure Analytics Administration Guide.

Process for manually creating a Log Source Extension

Alternatively, to manually create a log source extension, complete the following steps:

  1. Ensure that a log source is created in JSA.

    Use Universal DSM for the log source type to collect events from a source when the log source type not listed as a JSA supported DSM.

    For JSA 2014.8 and later, you don't need to use the Universal DSM to create a new log source type. If you want, you can use the DSM Editor only to create the new log source type, and then you manually create the log source. You can attach an LSX to a supported log source type, such as Windows, Bluecoat, Cisco, and others that are listed as JSA supported DSMs.

  2. To determine what fields are available, use the Log Activity tab to export the logs for evaluation.

  3. Use the extension document example template to determine the fields that you can use.

    It is not necessary to use all of the fields in the template. Determine the values in the log source that can be mapped to the fields in extension document template.

  4. Remove any unused fields and their corresponding Pattern IDs from the log source extension document.

  5. Upload the extension document and apply the extension to the log source.

  6. Map the events to their equivalents in the QIDmap.

    This manual action on the Log Activity tab is used to map unknown log source events to known JSA events so that they can be categorized and processed.

Building a Universal DSM

The first step in building a Universal DSM is to create the log source in JSA. When you create the log source, it prevents the logs from being automatically classified and you can export the logs for review.

  1. On the Admin tab, click the Log Sources icon.
  2. Click Add.
  3. Specify the name in the Log Source Name field.
  4. From the Log Source Type list, select Universal DSM.

    You might not see the Log Source Extension unless you already applied a log source extension to the JSA console

  5. From the Protocol Configuration list, specify the protocol that you want to use.

    This method is used by JSA to get the logs from the unsupported log source.

  6. For the Log Source Identifier, enter either the IP address or host name of the unsupported log source.
  7. Click Save to save the new log source and close the window.
  8. From the Admin tab, click Deploy Changes.

Exporting the Logs

Export the logs that are created after you build a Universal DSM

Typically you want a significant number of logs for review. Depending on the EPS rate of the unsupported log source, it might take several hours to obtain a comprehensive log sample.

When JSA can't detect the log source type, events are collected, but are not parsed. You can filter on these unparsed events and then review the last system notification that you received. After you reviewed the system notification, you can create a search that is based on that time frame.

  1. To look at only the events that are not parsed, filter the logs.
    1. Click the Log Activity tab.

    2. Click Add Filter.

    3. Select Event is Unparsed.

      Tip

      Type inside the Parameter text box to see the Event is Unparsed item.

    4. Select a time frame.

    5. If you see Information events from system notifications, right-click to filter them out.

    6. Review the Source IP column to determine what device is sending the events.

      You can view the raw event payloads. Typically, manufacturers put identifiable product names in the headers, so you can set your search to Display: Raw Events to show the payloads without having to manually open each event. Sorting by network can also help you find a specific device where the event originated from.

  2. Create a search for exporting the logs.
    1. From the Log Activity tab, select Search >Edit Search.

    2. For the Time Range, specify as enough time, for example 6 hours, from when the log source was created.

    3. Under Search Parameters, from the Parameter list, select Log Source (Indexed), from the Operator list, select Equals, and from the Log Source Group list, select Other, specify the log source that was created when you built the Universal DSM.

      Note

      Depending on your settings, you might see Log Source in the Parameter list instead of Log Source (Indexed).

    4. Click Search to view the results.

  3. Review the results in the console to check the payload.
  4. Optionally, you can export the results by clicking select Actions >Export to XML > Full Export (All Columns).

    Don't select Export to CSV because the payload might be split across multiple columns, therefore making it difficult to find the payload. XML is the preferred format for event reviews.

    1. You are prompted to download a compressed file. Open the compressed file and then open the resulting file.

    2. Review the logs.

      Event payloads are between the following tags:

      The following code shows an example payload:

      A critical step in creating a Universal DSM is reviewing the logs for usability. At a minimum, the logs must have a value that can be mapped to an event name. The event name must be a unique value that can distinguish the various log types.

      The following code shows an example of usable logs:

      The following codes shows an example of slightly less usable logs:

Common Regular Expressions

Use regular expressions to match patterns of text in the log source file. You can scan messages for patterns of letters, numbers, or a combination of both. For example, you can create regular expressions that match source and destination IP addresses, ports, MAC addresses, and more.

The following codes shows several common regular expressions:

The escape character, or "\", is used to denote a literal character. For example, "." character means "any single character" and matches A, B, 1, X, and so on. To match the "." characters, a literal match, you must use "\."

Table 1: Common Regex Expressions

Type

Expression

Type

\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}

IP Address

\d{1,5}

Port Number

(?:[0-9a-fA-F]{2}\:){5}[0-9a-fA-F]{2}

Protocol

(TCP|UDP|ICMP|GRE)

Device Time

\w{3}\s\d{2}\s\d{2}:\d{2}:\d{2}

Whitespace

\s

Tab

\t

Match Anything

.*?

Tip

To ensure that you don't accidentally match another characters, escape any non-digit or non-alpha character.

Building Regular Expression Patterns

To create a log source extension, you use regular expressions (regex) to match strings of text from the unsupported log source.

The following example shows a log entry that is referenced in the steps.

  1. Visually analyze the unsupported log source to identify unique patterns.

    These patterns are later translated into regular expressions.

  2. Find the text strings to match.Tip

    To provide basic error checking, include characters before and after the values to prevent similar values from being unintentionally matched. You can later isolate the actual value from the extra characters.

  3. Develop pseudo-code for matching patterns and include the space character to denote the beginning and end of a pattern.

    You can ignore the quotes. In the example log entry, the event names are DROP, PASS, and REJECT. The following list shows the usable event fields.

    • EventName: " kernel: VALUE "

    • SourceMAC: " MAC=VALUE "

    • SourceIp: " SRC=VALUE "

    • DestinationIp: " DST=VALUE "

    • Protocol: " PROTO=VALUE "

    • SourcePort: " SPT=VALUE "

    • DestinationPort: " DPT=VALUE "

  4. Substitute a space with the \s regular expression.

    You must use an escape character for non-digit or non-alpha characters. For example, = becomes \= and : becomes \:.

  5. Translate the pseduo-code to a regular expression.

    Table 2: Translating Pseudo-code to Regular Expressions

    Field

    Pseudo-code

    Regular expression

    EventName

    " kernel: VALUE

    "

    \skernel\:\s.*?\s

    SourceMAC

    " MAC=VALUE "

    \sMAC\=(?:[0-9a-fA-F]{2}\:){5}[0-9a-fA-F]{2}\s

    SourceIP

    " SRC=VALUE "

    \sSRC\=\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\s

    DestinationIp

    " DST=VALUE "

    \sDST\=\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\s

    Protocol

    " PROTO=VALUE "

    \sPROTO\=(TCP|UDP|ICMP|GRE)\s

    SourcePort

    " SPT=VALUE "

    \sSPT\=\d{1,5}\s

    DestinationPort

    " DPT=VALUE "

    \sDPT\=\d{1,5}\s

  6. Specify capture groups.

    A capture group isolates a certain value in the regular expression.

    For example, in the SourcePort pattern in the previous example, you can't pass the entire value since it includes spaces and SRC=<code>. Instead, you specify only the port number by using a capture group. The value in the capture group is what is passed to the relevant field in JSA.

    Insert parenthesis around the values you that you want capture:

    Table 3: Mapping Regular Expressions to Capture Groups for Event Fields

    Field

    Regular expression

    Capture group

    EventName

    \skernel\:\s.*?\s

    \skernel\:\s(.*?)\s

    SourceMAC

    \sMAC\=(?:[0-9a-fA- F]{2}\:){5}[0-9a-fA-F]{2}\s

    \sMAC\=((?:[0-9a-fA- F]{2}\:){5}[0-9a-fA-F]{2})\s

    SourceIP

    \sSRC\=\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\s

    \sSRC\=(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\s

    Destination IP

    \sDST\=\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\s

    \sDST\=(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\s

    Protocol

    \sPROTO\=(TCP|UDP|ICMP|GRE)\s

    \sPROTO\=((TCP|UDP|ICMP|GRE))\s

    SourcePort

    \sSPT\=\d{1,5}\s

    \sSPT\=(\d{1,5})\s

    DestinationPort

    \sDPT\=\d{1,5}\s

    \sDPT\=(\d{1,5})\s

  7. Migrate the patterns and capture groups into the log source extensions document.

    The following code snippet shows part of the document that you use.

Uploading Extension Documents to JSA

You can create multiple extension documents and then upload them and associated them to various log source types. The logic from the log source extension (LSX) is then used to parse the logs from the unsupported log source.

Extension documents can be stored anywhere before you upload to JSA.

  1. On the Admin tab, click Log Source Extensions.
  2. Click Add.
  3. Assign a name.
  4. If you are using the Universal DSM, don't select the extension document as the default for a Log Source Type.

    By selecting the Universal DSM as the default, it affects all associated log sources. A Universal DSM can be used to define the parsing logic for multiple custom and unsupported event sources.

  5. Optional: If you want to apply this log source extension to more than one instance of a log source type, select the log source type from the available Log Source Type list and click the add arrow to set it as the default.

    Setting the default log source type applies the log source extension to all events of a log source type, including those log sources that are automatically discovered.

    Ensure that you test the extension for the log source type first to ensure that the events are parsed correctly.

  6. Click Browse to locate the LSX that you saved and then click Upload.

    JSA validates the document against the internal XSD and verifies the validity of the document before the extension document is uploaded to the system.

  7. Click Save and close the window.
  8. Associate the log source extension to a log source.
    1. From the Admin tab, click Data Sources >Log Sources.

    2. Double-click the log source type that you created the extension document for.

    3. From the Log Source Extension list, select the document that you created.

    4. Click Save and close the window.

You can create multiple extension documents and then upload them and associated them to various log source types. The logic from the log source extension (LSX) is then used to parse the logs from the unsupported log source.

Extension documents can be stored anywhere before you upload to JSA.