Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

 

Configure Data Summarization

 

Data summarization refers to the process of creating a concise version of raw data and field data. Data can be summarized as a function of time or when a change occurs. You can improve the performance and disk space utilization of the Paragon Insights (formerly HealthBot) time series database (TSDB) by configuring data summarization methods to summarize the raw data and field data collected by Paragon Insights.

Paragon Insights collects data by using push or pull data collection methods. You can create Paragon Insights rules or use the available predefined rules to determine how and when data is collected. This collected telemetry data provides information about the state of network devices and its components. For more information on data collection methods, see Paragon Insights Data Ingest Guide.

You can create a raw data summarization profile to improve the performance and disk space utilization of the TSDB. Starting with Paragon Insights Release 4.0.0, you can create a rollup summarization profile to summarize processed data that is stored in fields in the TSDB. Field data is processed data that is stored in fields in the TSDB. A field is a single piece of information that forms a record in a database. In TSDB, multiple fields of processed data make a record. In releases earlier than Paragon Insights Release 4.0.0, only raw data can be summarized.

Table 1 provides a list of the supported data summarization algorithms and a description of their output:

Table 1: Descriptions of the Data Summarization Algorithms

Algorithm

Description of output

Latest

Value of the last data point collected within the time span.

Count

Total number of data points collected within the time span.

Mean

Average value of the data points collected within the time span.

Min

Minimum value of the data points collected within the time span.

Max

Maximum value of the data points within the time span.

On-change

Value of the data point whenever the value is different from the previous data point (occurs independently from the user-defined time span).

Stddev

Standard deviation of the data points collected within the time span.

Sum

Sum of the data points collected within the time span.

If no summarization algorithm is associated with the data, the following algorithms are used by default:

Data type

Data summarization algorithm

Float, integer, unsigned

Mean

Boolean, string

On-change

You can use data summarization profiles to apply specific summarization algorithms to raw data and field data collected by Paragon Insights for a specific device group:

These topics provide instructions on how to create a data summarization profile.

After you have created a data summarization profile, you can apply the profile to a device group. For more information, see Applying Data Summarization Profiles to a Device Group.

Creating a Raw Data Summarization Profile

To create a raw data summarization profile that can be applied to a device group:

  1. Click Settings > Summarization Profiles link in the left-nav bar.
  2. Select Raw Data from the Summarization Profiles list.

    The Raw Data Summarization Profiles page is displayed.

  3. Click (+) icon to add a summarization profile.

    The Add Raw Data Summarization Profile page is displayed.

  4. In the Name text box, enter the name of the profile.
  5. Click Add Type Aggregate to add an aggregate type.

    The Name and Function drop-down lists are displayed.

    Follow these steps to select a name data type and associate it with a data summarization algorithm.

    The algorithm configured for a specific sensor path name overrides the algorithm configured for the corresponding data type.

    1. Select a name data type from the Name drop-down list.

      The available name data types to choose from are string, integer, boolean, float, and unsigned integer.

      Starting in Paragon Insights Release 4.2.0, you can also select unsigned integer as a name data type. An unsigned integer is a data type that can contain values from 0 through 4,294,967,295.

    2. After you have selected a name data type, you associate it with a data summarization function.

      To associate a name data type with a data summarization function, select a function from the Functions drop-down list.

      The available functions to choose from are latest, count, mean, min, max, on-charge, stddev, and sum.

    3. (Optional) To add another aggregate type, click Add Type Aggregate, and repeat step a and step b.
  6. Click Add Path Aggregate to add an aggregate path.

    The Name and Function drop-down lists are displayed.

    To assign a sensor path name and associate it with a data summarization algorithm:

    Note

    The algorithm configured for a specific sensor path name overrides the algorithm configured for the corresponding data type.

    1. Enter a sensor path name in the Name text box.

      You can enter a path name for a sensor that is not supported by Paragon Insights. For sensors supported by Paragon Insights, the path name must be entered in the following format:

      Sensor

      Path Name Format

      Example

      Open Config

      sensor-path

      /components/component/name

      Native GPB

      sensor-name:sensor-path

      jnpr_qmon_ext:queue_monitor_element_info.percentage

      iAgent

      yaml-table-name:sensor-path

      REutilizationTable:15_min_cpu_idle

      SNMP

      snmp-table-name:sensor-path

      .1.3.6.1.2.1.2.2:jnxLED1Index

      ospfNbrTable:ospfNbrIpAddr

      Syslog

      pattern-set: sensor-path

      interface_link_down:operational-status

      Flow (NetFlow)

      template-name:sensor-path

      hb-ipfix-ipv4-template:sourceIPv4Address

    2. After you have entered a sensor path name, you associate it with a data summarization function.

      To associate a sensor path name with a data summarization function, select a function from the Functions drop-down list.

      The available functions to choose from are latest, count, mean, min, max, on-charge, stddev, and sum.

    3. (Optional) To add another aggregate path, click Add Path Aggregate, and repeat step a and step b.
  7. Click Save to only save the configuration.

    Click Save & Deploy to save and immediately deploy the configuration.

  8. You can now apply the raw data summarization profile that you created to a specific device group. For more information, see Applying Data Summarization Profiles to a Device Group.

Creating a Data Rollup Summarization Profile

Paragon Insights Release 4.0.0 supports data rollup summarization. You can create a rollup summarization profile to summarize processed data that is stored in fields in the TSDB. Field data is processed data that provides information on network devices and its components, and is stored in fields in the TSDB. A field is a single piece of information that forms a record in a database. In TSDB, multiple fields of processed data make a record. Data rollup summarization enables efficient data storage and also ensures retaining of data for a longer duration.

You can create a data rollup summarization profile to apply to a device group from the:

  • Paragon Insights graphical user interface (GUI)

  • Command line interface (CLI)

Create a Data Rollup Summarization Profile by using Paragon Insights UI

To create a data rollup summarization profile:

  1. Click Settings > Summarization Profiles link in the left-nav bar.
  2. Select Data Rollup from the Summarization Profiles list.

    The Data Rollup Summarization Profiles page is displayed.

  3. Click (+) icon to add a summarization profile.

    The Add Data Rollup Summarization Profile page is displayed.

  4. Enter the name of the profile in the Name text box.

    The maximum length is 64 characters.

    Regex pattern: “[a-zA-Z][a-zA-Z0-9_-]*

  5. Click Add Rule to add an existing Paragon Insights rule for which rollup summarization must be applied.

    The Name and Apply on Existing Data drop-down lists are displayed.

    Follow these steps to select a rule, and to apply the rule to the profile. You can also apply the rule to existing data.

    1. Select an existing Paragon Insights from the Name drop-down list.
    2. To apply the rule that you selected to existing data, select True from the Apply on Existing Data drop-down list.

      The default value is False.

    3. Click Add Field to define fields of the rule configured for which data rollup summarization must be applied.

      To associate a field to an aggregate function:

      1. Select a field from the Name drop-down list for which data must be aggregated.
      2. Select one or more aggregate functions from the Aggregate Function drop-down list that you want to apply to a field.
    4. (Optional) To add another rule to the profile, click Add Rule, and repeat step a through step c.
  6. Click Add Data Rollup Order to define the frequency at which rollup summarization should occur.

    The Name and Retention Policy drop-down lists, and the Rollup Interval text box are displayed.

    To define a data rollup order:

    1. Enter a name to identify the data rollup order in the Name text box.

      The maximum length is 64 characters.

      Regex pattern: “[a-zA-Z][a-zA-Z0-9_-]*

    2. Enter a value in the Rollup Interval text box to define the interval in which data is summarized.

      Regex pattern: “[1-9][0-9]*[mhdw]”, where m is minutes, h is hours, d is days, and w is weeks.

      Minimum value is 30m. Maximum value is 52w.

    3. Select the retention policy for the rollup order from the Retention Policy drop-down list. A retention policy defines how long you want to retain the rolled-up data.

      Selecting a retention policy is optional. If you do not select a retention policy, the device group retention policy is considered by default.

    4. (Optional) To define another data rollup order, click Add Data Rollup Order, and repeat step a through step c.
  7. Click Save to only save the configuration.

    Click Save and Deploy to save and immediately deploy the configuration.

  8. You can now apply the data rollup summarization profile that you created to a specific device group. For more information, see Applying Data Summarization Profiles to a Device Group.

Create a Data Rollup Summarization Profile by using CLI

Figure 1 is an example configuration of how you can configure a data rollup summarization profile from the CLI.

Figure 1: Example CLI Configuration of Creating a Data Rollup Summarization Profile
Example CLI Configuration of
Creating a Data Rollup Summarization Profile

Applying Data Summarization Profiles to a Device Group

After you create a data summarization profile, you can apply the profile to a specific device group to start summarizing TSDB data:

  1. Click Configuration > Device Group in the left-nav bar.

    The Device Group Configuration page is displayed.

  2. Select the check box next to the name of the device group to which you want to apply the data summarization profile.
  3. Click the Edit Device Group icon to edit the device group.

    The Edit <device-group-name> page is displayed.

  4. Apply a raw data summarization profile.

    To apply a raw data summarization profile to a device group:

    1. Click Summarization.

      The Time Span and Data Summarization text boxes are displayed.

    2. Enter the Time Span in seconds (s), minutes (m), hours (h), days (d), weeks (w), or years (y).
    3. Choose the data summarization profiles from the drop-down list to apply the ingest data. To edit or view details about saved data summarization profiles, go to the Data Summarization page and click the Settings menu option in the left-nav bar.

      If you select two or more profiles, the following guidelines apply:

      • If the same data type or sensor path name is configured in two or more profiles, the associated algorithms will be combined.

      • The table that stores the summarization output includes columns of summarized data for each algorithm associated with each data field collected by Paragon Insights. The naming convention for each column is as follows:

        Number of algorithms associated with a data field

        Column name for the summarized output

        1

        field-name

        Example: 5_sec_cpu_idle

        2

        field-name_first-algorithm-name, field-name_ second-algorithm-name

        Example: 5_sec_cpu_idle_MIN, 5_sec_cpu_idle_MAX

        3

        field-name_first-algorithm-name, field-name_ second-algorithm-name, field-name_ third-algorithm-name...

    Apply a data rollup summarization profile

    Points to remember before you apply a data rollup summarization profile to a device group:

    • Ensure that the rules present in the rollup profile are already associated with the device group.

    • You can add one or more than one rollup summarization profile to a device group.

    • Rules configured across all the profiles associated with the device group must be unique.

    • While associating a rollup profile with a device group, the interval of the first data rollup order must be less then the device group retention policy to avoid data overflow. The device group retention policy is set to 7 days by default.

    • When you want to remove a rule that is associated to a device group, you must first remove the data rollup summarization profile.

    To apply a data rollup summarization profile to a device group:

    1. Click Rollup Summarization.

      The Rollup Summarization Profiles drop-down list is displayed.

    2. Select the rollup summarization profiles you want to associate to this device group from the Rollup Summarization Profiles drop-down list.
    3. (Optional) You can also deploy rollup configuration at the device group-level by using the CLI.

      See Figure 2 for an example CLI configuration.

      Figure 2: Example CLI Configuration
      Example CLI Configuration
  5. Click Save to only save the configuration.

    Click Save and Deploy to save and immediately deploy the configuration.

Release History Table
Release
Description
Paragon Insights Release 4.0.0 supports data rollup summarization. You can create a rollup summarization profile to summarize processed data that is stored in fields in the TSDB.