Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

 

QRadar Pulse Dashboard Items

 

Create dashboard items to include in one or more of your dashboards. You can see only the dashboard items and dashboards that you create in your workspace. However, you can share them with others by exporting them or by opening a dashboard or dashboard item onto a shared monitor, like a SOC wall.

Creating Dashboard Items from an Offense Data Source

Use the offense API endpoints as a data source for your dashboard items.

Administrators can't create dashboard items or dashboards for other users, but they can share their own dashboards with other users.

To query the offenses data source (/siem/offenses endpoint), you must have the Offenses permission. The data that is returned is restricted based on your security profile.

  1. Click Manage Dashboard Items >New Dashboard Item.
  2. On the New Dashboard Item page, enter a name and a description for the item.
  3. Select Offense from the list in the Query section, and select the API fields that you want to view in the results from the Fields list. Use the Filter and Sort as required to fine-tune the results.

    The following image shows an example of completed API parameters.

  4. Pick a refresh time for how often you poll the data source.

    Choose a refresh rate that is greater than the selected query time. Each time the dashboard item is refreshed, the REST API runs, so methods have short response times. The current default refresh rate for dashboard items is every minute.

  5. Click Run Query.

    When you first create the item, you can't configure the charts when no data results are returned. Try making the criteria in the fields less strict and run the query again.

  6. Create a dashboard chart in the Views section.

    Because you can create multiple views from the same query, give the view a unique name. By default, the chart's title and status on the title bar are displayed; to hide them, click the More options icon and switch the settings to off.

  7. Select a chart type and complete the corresponding fields for the chart. For use cases to help you decide which chart type to use, see Dashboard Item Chart Types.

    Chart type

    Bar

    Big Number

    Pie

    Scatter

    Tabular

  8. Preview how the chart looks and then click Save.

You can edit a dashboard item and save it without rerunning the query. For example, if a query doesn't return results, such as when the time period isn't long enough to pick up new events, or if the magnitude or severity value isn't applicable when you run the query, you can save the dashboard item. If you edit the query, you must run the query again before you can save the dashboard item.

Deleting a dashboard item removes it from all of the dashboards it belongs to. If the deleted dashboard contains parameters, the parameters are not deleted from QRadar Pulse.

Creating Dashboard Items from an AQL Data Source

You can use Ariel Query Language (AQL) statements to create dashboard items. AQL is a structured query language that you use to extract, filter, and manipulate event and flow data that you extract from the Ariel database in QRadar.

Read the following topics in the Juniper Secure Analytics Ariel Query Language Guide to learn about the Ariel Query Language that you use in QRadar Pulse to create dashboard items based on AQL data sources:

  • Within the overview section, several topics explain the statements and clauses you use. The statements and clauses appear as keywords in QRadar and QRadar Pulse when you enter them in the query field.

  • Use AQL to retrieve specific fields from the events, flows, and simarc tables in the Ariel database.

Administrators can't create dashboard items or dashboards for other users, but they can share their own dashboards with other users.

  1. Click Manage Dashboard Items >New Dashboard Item.
  2. On the New Dashboard Item page, enter a name and a description for the item.
  3. Select AQL from the list in the Query section, and enter an AQL statement. For more information, see QRadar Pulse Dashboard Items.
    1. Insert existing parameters in the statement. Click the Insert Parameter icon, and then click Insert for each relevant parameter.

    2. To change the default value of the parameter, click the View Parameters icon, and click Save after you set the default value.

      When you change the default value for a parameter, you're changing the value everywhere the parameter is used in your workspace, except in expanded or pinned dashboards and dashboard items. If you don't set the value as the default value, the updated change applies only to the current session. However, if you set the value as the default, the current session value also uses that value.

    3. To add a parameter to your workspace, click Add, give the parameter a name and default value, if needed, and then click Save.

      Note

      After you add parameters to a dashboard item on a dashboard for the first time, the Parameters card appears on the dashboard. If you remove parameters from the dashboard item, and no other item in that dashboard uses the parameter, the Parameters card disappears.

  4. Pick a refresh time for how often you poll the data source. Choose a refresh rate that is greater than the selected query time. Each time the dashboard item is refreshed, the query reruns, so methods have short response times. The current default refresh rate for dashboard items is every minute. The timer for the refresh rate begins after the query is completed.

    For example, if the refresh rate is every minute, and the query takes 3 minutes to complete, the refresh rate starts only after the 3-minute run ends.

  5. Click Run Query.

    When you first create the item, you can't configure the charts when no data results are returned. Try making the criteria in the fields less strict and run the query again.

    If your AQL query contains parameters without any values, enter them on the Parameters page. You must enter a value for each parameter so that the query runs successfully. If the query is successful, the results are displayed next to the statement.

  6. Create a dashboard chart in the Views section.

    Because you can create multiple views and charts from the same query, give the view a unique name. By default, the chart's title and status on the title bar are displayed; to hide them, click the More options icon and switch the settings to Off.

  7. Select a chart type and configure the relevant properties. For use cases to help you decide which chart type to use, see Dashboard Item Chart Types.

    Chart type

    Bar

    Big Number

    Geographic

    Pie

    Scatter

    Tabular

    Time Series

  8. Preview how the chart looks and then click Save.Tip

    The labels for the chart come from the queries that are used. If they are unintelligible in the preview, edit the labels in the View section.

You can edit a dashboard item and save it without rerunning the query. For example, if a query doesn't return results, such as when the time period isn't long enough to pick up new events, or if the magnitude or severity value isn't applicable when you run the query, you can save the dashboard item. If you edit the query, you must run the query again before you can save the dashboard item.

Deleting a dashboard item removes it from all of the dashboards it belongs to. If the deleted dashboard contains parameters, the parameters are not deleted from QRadar Pulse.

Tips for Creating AQL Queries for Dashboard Charts

Tips to help you create AQL queries and dashboard charts more easily cover aliases, time and date formats, and transformation lookup functions.

  • Any AQL statement that doesn't contain a time criteria such as start, stop, or last runs only for 5 minutes.

  • Don't use a transformation lookup function in an IN operator in a WHERE clause. The IN operator specifies multiple values in the WHERE clause, but can cause performance issues when you run the query. For example, the following query can cause performance issues:

    Where logsourcetypename(deviceType) IN ('a','b')

  • Always use aliases to ensure that your field names don't vary from one QRadar version to another. For example, in the following query, Active Offense Sum is an alias for the SUM_Active Offense Count field.

    select ("SUM_Active Offense Count" / 2) as 'Active Offense Sum', ("SUM_Dormant Offense Count" / 2) as 'Dormant Offense Sum', "Time" * 1000 as 'sTime' from GLOBALVIEW('Offenses Over Time','NORMAL') order by "Time" desc last 2 days

  • Use the millisecond time format in the Time in the AQL query; for example, starttime-starttime%60000.

  • Use proper date and year formats in your queries, especially in a GLOBALVIEW. Dates are not automatically appended; for example, use StartTime, 'YYYY-MM-dd HH:mm'.

  • Go to the Log Activity or Network Activity tab and run an AQL search to get auto-completion and messaging. If you're satisfied with the results you get there, then copy and paste the statement in the Query section in QRadar Pulse. The GLOBALVIEW source can be used only in REST API calls.

  • See the following topics in the Juniper Secure Analytics Ariel Query Language Guide:

    • Overview of Ariel Query Language

    • AQL logical and comparison operators

    • System performance query examples

    • Events and flows query examples

    • Reference data query examples

    • User and network monitoring query examples

Sample Event, Log Source, and Storage Usage AQL Statements

Use the following examples to monitor events, log sources, and storage usage or you can edit the queries to suit your requirements.

  • Log source summary

  • Unique events

  • Unparsed events percentage

  • Unparsed events percentage for a specific log source type

  • Number of partial matches per rule

  • Number of partial matches per event processor

  • Number of partial matches per rule and per event processor

  • Storage consumption per log source type

  • Storage usage

  • CPU usage

  • Memory usage

  • Expensive CEPs, log sources, and rules

  • Expensive searches

  • Executed AQL searches by user

Log Source Summary

Provides a list of unique log source types, including the number of log sources, EPS, and the percentage of unparsed events.

SELECT LOGSOURCETYPENAME(devicetype) AS "LogSourceType",UNIQUECOUNT(logsourceid) as "Number of Log Sources", COUNT(*)/3600 as "EPS",(DOUBLE(COUNT(isunparsed)) / COUNT(*)) * 100 AS "PercentUnparsed" FROM events GROUP BY devicetype ORDER BY "PercentUnparsed" DESC LAST 1 HOURS

Unique Events

Provides a list of unique events and the log source type, QID, high-level category, low-level category, and the event count.

SELECT LOGSOURCETYPENAME(devicetype) AS "Log Source Type", QIDNAME(qid) AS "Event Name", qid as "QID", CATEGORYNAME(highlevelcategory) AS "High-level Category", CATEGORYNAME(category) AS "Low-level Category", LONG(COUNT(*)) as "Number of Events" FROM events GROUP BY qid, devicetype ORDER BY COUNT(*) DESC LAST 1 HOURS

Unparsed Events Percentage

Provides the percentage of events that are unparsed for each log source type. Any log sources that are greater than 20% unparsed must be addressed.

SELECT LOGSOURCETYPENAME(devicetype) AS "Log Source Type", devicetype AS "Log Source ID", LONG(UNIQUECOUNT(logsourceid)) as "Number of Log Sources", LONG(SUM(eventcount)) as "Total Events", LONG(COUNT(*)) as "Aggregated Events", LONG(COUNT(isunparsed)) AS "Unparsed Events", STR(LONG(SUM(eventcount)/24*3600)) as "EPS", LONG("Unparsed Events"*100) / "Total Events" AS "PercentUnparsed" FROM events GROUP BY devicetype ORDER BY "PercentUnparsed","Unparsed Events" DESC LAST 24 HOURS

Unparsed Events Percentage for a Specific Log Source Type

To further investigate the parsing issues with one specific log source type (for example, devicetype=11), you can run the following query that returns a breakdown of the statistics per log source.

SELECT LOGSOURCENAME(logsourceid) AS "Log Source Name", LONG(SUM(eventcount)) as "Total Events", LONG(COUNT(*)) as "Aggregated Events", LONG(COUNT(isunparsed)) AS "Unparsed Events", STR(LONG(SUM(eventcount)/24*3600)) as "EPS", LONG("Unparsed Events"*100) / "Total Events" AS "PercentUnparsed" FROM events WHERE GROUP BY logsourceid ORDER BY "PercentUnparsed","Unparsed Events" DESC LAST 24 HOURS

Tip

Add more fields as required for your environment. Create an AQL parameter for the devicetype=11 variable. For more information, see QRadar Pulse Dashboard Items.

Number Of Partial Matches Per Rule

Provides a list of all rules and building blocks with the number of partial matches within the specified time period.

SELECT RULENAME(partialmatchlist) as "Rule Name", LONG(COUNT(*)) as "Number of Partial Matches" FROM events WHERE partialmatchlist IS NOT NULL GROUP BY "Rule Name" ORDER BY "Number of Partial Matches" DESC LAST 1 HOURS

Number Of Partial Matches Per Event Processor

Provides a list of all rules and building blocks with the number of partial matches within the specified time period.

SELECT HOSTNAME(processorid) as "Event Processor Name", LONG(COUNT(*)) as "Number of Partial Matches" FROM events WHERE partialmatchlist IS NOT NULL GROUP BY "Event Processor Name" ORDER BY "Number of Partial Matches" DESC LAST 1 HOURS

Number Of Partial Matches Per Rule and Per Event Processor

Provides a list of all rules and building blocks, including the number of partial matches per rule within the specified time period.

SELECT HOSTNAME(processorid) as "Event Processor Name", RULENAME(partialmatchlist) as "Rule Name", LONG(COUNT(*)) as "Number of Partial Matches" FROM events WHERE partialmatchlist IS NOT NULL GROUP BY "Event Processor Name", "Rule Name" ORDER BY "Number of Partial Matches" DESC LAST 1 HOURS

Storage Consumption Per Log Source Type

Provides a breakdown of storage that is used by each log source type during 1 day.

Note

This query doesn't include the storage that is used for QRadar indexes. However, most indexes are equally distributed among all events regardless of the log source type or the size of the payload.

SELECT LOGSOURCETYPENAME(deviceType) AS LogSource, LONG(MIN(STRLEN(UTF8(payload)))) AS "Minimum Payload Size (Bytes)", LONG(MAX(STRLEN(UTF8(payload)))) AS "Maximum Payload Size (Bytes)", LONG(AVG(STRLEN(UTF8(payload)))) AS "Average Payload Size (Bytes)", LONG(STDEV(STRLEN(UTF8(payload)))) AS "Standard Deviation (Bytes)", LONG(COUNT(logsourceid)) AS EventCount, LONG(EventCount * "Average Payload Size (Bytes)") / (1024 * 1024) as "Total Storage (MB)" FROM events GROUP BY deviceType ORDER BY "Total Storage (MB)" DESC LAST 24 HOURS

Storage Usage

Provides a daily summary of the amount of storage that is used on all QRadar appliances.

SELECT DATEFORMAT(starttime, 'yyyy-MM-dd') as "Date", "Hostname" as "QRadar Appliance Name", LONG(MAX("Value")/(1024*1024*1024)) as "Storage Used (GB)" FROM events WHERE (qid = 94000001) AND ((LONG(starttime/1000)%(24*3600)) < 20) AND ("Metric ID" = 'DiskSpaceUsed') AND (Element = '/store') GROUP BY "Date", "Hostname" ORDER BY "Date", "Hostname"

Tip

Run the search for a minimum of 1 day to return results. To improve results, index the Metric ID custom property.

CPU Usage

Provides a break-down of CPU time that is used by each component on every QRadar appliance

SELECT DATEFORMAT(starttime, 'yyyy-MM-dd') as "Date", "Hostname" as "QRadar Appliance Name", "Component Type", LONG(SUM("Value")) as "CPU Time" FROM events WHERE (qid = 94000001) AND ("Metric ID" = 'ProcessCPUTime') GROUP BY "Date", "Hostname", "Component Type" ORDER BY "Date", "Hostname", "Component Type"

Memory Usage

Provides a break-down of heap memory that is used by each component on every QRadar appliance.

SELECT DATEFORMAT(starttime, 'yyyy-MM-dd') as "Date", "Hostname" as "QRadar Appliance Name", "Component Type", LONG(SUM("Value")/(1024*1024*1024)) as "Memory Usage Per Day (GB)" FROM events WHERE (qid = 94000001) AND ("Metric ID" = 'HeapMemoryUsed') GROUP BY "Date", "Hostname", "Component Type" ORDER BY "Date", "Hostname", "Component Type"

Expensive CEPs, Log Sources, and Rules

Provides a complete list of expensive complex event processing (CEP), log sources, and rules that are identified by QRadar.

Note

This statement uses the following custom properties from QRadar Development Intelligence app: expensivecp, expensivelogsource, and expensiverules.

SELECT DATEFORMAT(starttime,'yyyy-MM-dd HH:mm:ss') as "timestamp",sourceip, "expensivecp","expensivelogsource","expensiverules", UTF8(payload) FROM events WHERE devicetype=147 AND (expensivecp is not NULL OR expensivelogsource is not NULL OR expensiverules is not NULL) ORDER BY timestamp DESC LAST 48 HOURS

Expensive Searches

Provides a list of searches that were run in the past 24 hours. The list is sorted based on the execution time.

Note

This statement uses the following custom properties from QRadar Development Intelligence app: expensivecp, expensivelogsource, and expensiverules.

SELECT "searchid", "searchpriority", "searchlimit", "searchtime" FROM events WHERE qid=28250295 ORDER BY searchtime DESC LAST 24 HOURS

Executed AQL Searches by User

Provides a list of all searches that were run by each user.

Note

This statement uses the following custom properties from QRadar Development Intelligence app: expensivecp, expensivelogsource, and expensiverules.

SELECT username, "Ariel Source", "Ariel Cursor ID", "searchpriority", "AQL Statement" FROM events WHERE qid=28250254 ORDER BY username LAST 2 HOURS

Copying Query Samples

If you copy and paste a query sample that contains single or double quotation marks, you must retype the quotation marks to be sure that the query parses.

Use Cases for AQL Parameters

Parameters help make it simple to reuse common elements in multiple AQL queries so that it's easier to create dashboard items and share them. You no longer need to create single queries for similar entities. Any query that uses the parameter can automatically use the value.

Use parameters to represent common entities, such as the IP of your console. For example, rather than typing the unique IP address into each query of every relevant chart, add the specific IP address into the Console_IP parameter. Any workspace dashboard items that use that parameter is instantly updated.

Search Events for a Specific User

You’re a senior analyst in a SOC and want to investigate some suspicious behavior of one of your users, Bob. Build a dashboard that captures information you need to care about when Bob does something malicious. For example, how many logins are associated with Bob in the last hour? Create parameters for username, events per user, time period, time value, sourceip, and destinationip.

Include dashboard items that cover the top event categories, top 10 log sources, location of events, events per user, and time of day. When you export and share the dashboard with others, they can replace Bob's user name with the user name of their choice.

Investigate Events During Specific Time Periods

In a time series chart, you see a large spike in the number of login failure events. Change the time period and run the query again to see when the spike started. For example, if you change the time period to the last 30 days instead of the last 4 days, was that when the spike started?

Sharing Dashboards with Others

Your team shares responsibility for managing several managed hosts around the world. You create a dashboard that monitors specific servers for various health metrics, such as top disk usage and disk usage distribution. Create a parameter for servername, and then share the dashboard with your colleagues. All that they need to do is add the server names that they monitor into the parameter values, and they get results specific to their servers.

Creating AQL Parameters for Your Workspace Dashboards

Create parameters that you can use in all of your dashboard items with an AQL data source, such as Time Period and Time Value. Using common parameters and default values can reduce the time that it takes to create or edit individual AQL queries for your dashboard items.

In the AQL statement, parameters appear within curly braces, as in the following example:

Because curly braces are also used in an AQL statement within a quoted string (a string literal between quotation marks), you must use the backward slash (\) as an escaping character before the opening curly brace. This syntax helps ensure that QRadar Pulse doesn't interpret the quoted string as a parameter and cause incorrect query results. If you have a backward slash in a quoted string, add an extra backward slash as the escaping character.

After you upgrade to QRadar Pulse V2.1.3, the text on the Parameter Actions and Export as JSON menus might appear as one word per line. Clear your browser cache to fix the issue.

  1. From a dashboard that contains AQL-based items, such as the Events and flow metrics dashboard, click the Parameter Actions icon in the toolbar and select Manage Parameters from the menu.

    The Manage Parameter Values page displays the parameters that are already created in your workspace. If any of the parameters have a default value, it is listed in the Default value column. The Number of occurrences column indicates the number of dashboard items that use each parameter. Hover over the number to see which dashboard items use the parameter.

  2. To set a default value for a parameter, click the More options icon, enter the default value in the field, and then click Save.
  3. To add a parameter to your workspace, click Add, give the parameter a name and default value, and then click Save.
  4. To delete a parameter, first make sure that it doesn't belong to any dashboard item by hovering over the number to see the names of each item. After you remove the parameter from those items, then you can click Delete.
  5. Click Back to Dashboard to return to the main view and continue your work.

Adding Parameters to Existing AQL-based Dashboard Items

In this example scenario, you create two parameters and add them to the Top 10 Log Sources dashboard item in the Events and flow metrics dashboard.

  1. On the Top 10 Log Sources dashboard item card, click the Edit Item icon.
  2. Click Insert >Add, and create two parameters: one called Time Period (with a value of "HOURS"), and one called Time Value (with a value of "2"). Click Save after you create each parameter. The parameters are available for all dashboard items in your workspace.
  3. To modify the query in the AQL Statement field, take the following steps:

    SELECT logsourcename(logsourceid) AS 'MY Log Sources', LONG(SUM(eventcount) / ((MAX(endTime) - MIN(startTime) + 1) / 1000.0)) as event_rate FROM events GROUP BY logsourceid ORDER BY event_rate DESC LIMIT 10 LAST 2 HOURS

    1. Replace "HOURS" by clicking the Insert Parameter icon, and then click Insert for the Time Period parameter.

    2. Replace "2" with Time Value, and then click Run Query. The updated query looks like the following example:

      SELECT logsourcename(logsourceid) AS 'MY Log Sources', LONG(SUM(eventcount) / ((MAX(endTime) - MIN(startTime) + 1) / 1000.0)) as event_rate FROM events GROUP BY logsourceid ORDER BY event_rate DESC LIMIT 10 LAST {Time Value} {Time Period}

  4. After the query runs successfully, click Save.

The new parameters are added to the Parameters card, and are prefaced by an asterisk (*) to indicate that they are default values for the workspace. Only parameters that are used by the items in this dashboard appear in the Parameters section.