Creating Reference Data Collections by Using the Command Line

Use the command line to manage reference data collections that cannot be managed in JSA, such as reference maps, map of sets, map of maps, and tables. Although it's easier to manage reference sets using JSA, use the command line when you want to schedule management tasks.

Use the ReferenceDataUtil.sh script to manage reference sets and other types of reference data collections.

When you use an external file to populate the reference data collection, the first non-comment line in the file identifies the column names in the reference data collection. Each line after that is a data record that gets added to the collection. While the data type for the reference collection values is specified when the collection is created, each key is an alphanumeric string.

The following table shows examples of how to format data in an external file that is to be used for populating reference maps.

Table 1: Formatting Data in an External File to Be Used for Populating Reference Data Collections
Type of reference collection	Data formatting examples
Reference map	`key1,data` `key1,value1` `key2,value2`
Reference map of sets	`key1,data` `key1,value1` `key1,value2`
Reference map of maps	`key1,key2,data` `map1,key1,value1` `map1,key2,value2`

You can also create reference data collections by using the /reference_data endpoint in the JSA RESTful API.

Using SSH, log in to JSA as the root user.
Go to the /opt/qradar/bin directory.
To create the reference data collection, type the following command:

./ReferenceDataUtil.sh create name [SET | MAP | MAPOFSETS | MAPOFMAPS | REFTABLE] [ALN | NUM | IP | PORT | ALNIC | DATE] [-timeoutType=[FIRST_SEEN | LAST_SEEN]] [-timeToLive=]
To populate the map with data from an external file, type the following command:

./ReferenceDataUtil.sh load name filename [-encoding=...] [-sdf=" ... "]

Here are some examples of how to use the command line to create different types of reference data collections:

Create an alphanumeric map:

./ReferenceDataUtil.sh create testALN MAP ALN
Create a map of sets that contains port values that will age out 3 hours after they were last seen:

./ReferenceDataUtil.sh create testPORT MAPOFSETS PORT -timeoutType=LAST_SEEN -timeToLive='3 hours'
Create a map of maps that contains numeric values that will age out 3 hours 15 minutes after they were first seen:

./ReferenceDataUtil.sh create testNUM MAPOFMAPS NUM -timeoutType=FIRST_SEEN -timeToLive='3 hours 15 minutes'
Create a reference table where the default format is alphanumeric:

./ReferenceDataUtil.sh create testTable REFTABLE ALN -keyType=ipKey:IP,portKey:PORT,numKey:NUM,dateKey:DATE

Log in to JSA to create rules that add data to your reference data collections. You can also create rule tests that detect activity from elements that are in your reference data collection.

Command Reference for Reference Data Utilities

You can manage your reference data collections by using the ReferenceDataUtil.sh utility on the command line. The following commands are available to use with the script.

Create
Update
Add
Delete
Remove
Purge
List
Listall
Load

Create

Creates a reference data collection.

name	The name of the reference data collection.
[SET \| MAP \| MAPOFSETS \| MAPOFMAPS \| REFTABLE]	The type of reference data collection.
[ALN \| ALNIC \| NUM \| IP \| PORT \| DATE]	The type of data in the reference set. ALN specifies alphanumeric values. This data type supports IPv4 and IPv6 addresses. ALNIC specifies alphanumeric values, but rule tests ignore the case. This data type supports IPv4 and IPv6 addresses. NUM specifies numeric values. IP specifies IP addresses. This data type supports only IPv4 address. PORT specifies port addresses. DATE specifies date values.
[-timeoutType=[FIRST_SEEN \| LAST_SEEN]]	Specifies whether the amount of time the data elements remain in the reference data collection is from the time the element was first seen or last seen.
[-TimeToLive='']	The amount of time the data elements remain in the reference data collection.
[-keyType=name:elementType,name:elementType,...]	A mandatory REFTABLE parameter of consisting of key name to ELEMENTTYPE pairs.
[-key1Label='']	An optional label for key1, or the primary key. A key is a type of information, such as an IP address.
[-valueLabel='']	An optional label for the values of the collection.

Update

Updates a reference data collection.

name	The name of the reference data collection.
[-timeoutType=[FIRST_SEEN \| LAST_SEEN]]	Specifies whether the amount of time the data elements remain in the reference data collection is from the time the element was first seen or last seen.
[-timeToLive='']	The amount of time the data elements remain in the reference data collection.
[-keyType=name:elementType,name:elementType,...]	A mandatory REFTABLE parameter of consisting of key name to elementType pairs.
[-key1Label='']	An optional label for key1.
[-valueLabel='']	An optional label for the values of the collection.

Add

Adds a data element to a reference data collection.

name

The name of the reference data collection.

The key value pair that you want to add. The keys are alphanumeric strings.

MAP and MAPOFSETS require Key 1.
MAPOFMAPS and REFTABLE require Key 1, and the second-level Key 2.

[-sdf=" ... "]

The Simple Date Format string that is used to parse the date data.

Delete

Deletes an element from a reference data collection.

name

The name of the reference data collection.

The key value pair that you want to add. The keys are alphanumeric strings.

MAP and MAPOFSETS require Key 1.
MAPOFMAPS and REFTABLE require Key 1, and the second-level Key 2.

[-sdf=" ... "]

The Simple Date Format string that is used to parse the date data.

Remove

Removes a reference data collection.

name	The name of the reference data collection.

Purge

Purges all elements from a reference data collection.

name	The name of the reference data collection.

List

Lists elements in a reference data collection.

name	The name of the reference data collection.
[displayContents]	Lists all elements in the specified reference data collection.

Listall

Lists all elements in all reference data collection.

[displayContents]

Lists all elements in all reference data collections.

Load

Populates a reference data collection with data from an external .csv file.

name	The name of the reference data collection.
filename	The fully qualified file name to be loaded. Each line in the file represents a record to be added to the reference data collection.
[-encoding=...]	Encoding that is used to read the file.
[-sdf=" ... "]	The Simple Date Format string that is used to parse the date data.