Processor: Extensible Service Data Collector

The Extensible Service Data Collector processor collects data supplied by a custom service that is not ‘lldp’, ‘bgp’ or ‘interface’.

Input Types - No inputs. This is a source processor.

Output Types - NSTS, DSSTS

Properties

Data Type
Type of data the service collects: numbers (ns) (such as device temperature), discrete states (dss) (such as device status), text or tables
Graph Query (graph_query)

One or more queries on graph specified as strings, or a list of such queries. (String will be deprecated in a future release.) Multiple queries should provide all the named nodes referenced by the expression fields (including additional_properties). Graph query is executed on the “operation” graph. Results of the queries can be accessed using the “query_result” variable with the appropriate index. For example, if querying property set nodes under name “ps”, the result will be available as “query_result[0][“ps”]”.

In collector processors (*_collector, if_counter) it is used to choose a set of nodes for further processing (for example, all leafs, or all interfaces between leaf and spines)

In other processors it is used for general parameterization and it is only supported as a list of queries.

Fabric Interfaces Example
   graph_query: "node("system", role="leaf", name="system").
                 out("hosted_interfaces").
                 node("interface", name="iface").out("link").
                 node("link", role="spine_leaf")"
Leafs and Spines using two queries Example
   graph_query: ["node("system", role="leaf", name="system")",
                 "node("system", role="spine", name="system")"]
Ingestion filter (ingestion_filter)

New (reserved) key. Ingestion filter determines what metrics from the collector make it into the probe. As of version 3.0, we support a degenerate case of ingestion filter, that is, probe specifies full identities of all metrics that need to be ingested. With this feature, you can ingest metrics that satisfy a criterion that is expressed using an ingestion filter.

Ingestion filter is authored by probe authors, evaluated by the controller component that is responsible for ingesting raw telemetry into stage outputs within the probes. It is also propagated as a collection filter to the telemetry collector plugins.

Keys available to express in the filter are same as the metric identity keys.

  • No metric identity key can exist directly under “properties”. If any metric identity key is mistakenly specified directly under properties a validation error is raised.
  • Any missing metric identity key under “ingestion_filter” is assumed to match.
  • Only explicitly specified keys under “ingestion_filter” can be referenced by the rest of the probe configuration. This is to enhance probe readability and allow better overall validation.
  • The data_type must be one of the table data types.
  • Existing reserved key “keys” is now made optional and can be omitted. The key names should exactly match those specified in the schema of the corresponding service definition.
Keys (keys)
List of keys that are significant for specifying data elements for this service
Query Group by (query_group_by)

List (of strings) of node and relationship names used in the graph query to group query results by. Each element in this list represents a named node or relationship matcher in the graph_query field.It is not an expression to be consistent with existing group_by field in grouping processors. Non-expression is simple and more intuitive.

When grouping is active (query_group_by is not null), query results are d by the specified list of names, where one output item is created per each group. In this case, the expressions can only access matcher names specified in query_group_by and the query results for each group are accessed using a new group_items variable. The group_items variable is a list of query results, where each result has named nodes/relationships, not present in query_group_by.

The following table describes the behavior for various values of this field:

Value of query_group_by field Semantics
Omitted or provided as json null (ala None in Python) No grouping is done. This is equivalent to current behavior of extensible_data_collector. Using ‘group_items’ in this case is not permitted and results in probe error state.
Empty list ([]) Produces one group containing all the query results.
One or more matcher names The query results are grouped by the specified nodes or relationships. If this list covers all available matchers in the query, the number of groups is equal to the number of query results.
Value Map

A mapping of discrete-state values to human readable strings. A dictionary with all possible Discrete-State-Set states mapped to human-readable representation; applicable for Discrete-State-Set data (that is, when data_type is ‘dss’) only.

Sample value map for interface status
{
    "0": "unknown",
    "1": "down",
    "2": "up",
    "3": "missing"
}
Service name (service_name)
Name of the custom collector service.
System ID
Expression mapping from graph query to a system_id, e.g. “system.system_id” if “system” is a name in the graph query.
Execution count
Number of times the data collection is done.
Service input (service_input)
Data to pass to telemetry collectors, if any. Can be an expression.
Service interval (service_interval)
Telemetry collection interval in seconds. Can be an expression.
Additional Keys
Each additional key/value pair is used to extend properties of output stages where value is considered as an expression executed in context of the graph query and its result is used as a property value with respective key. The value of this property is evaluated for each item to associate items with metrics provided by a corresponding collector service. The association is done by keys because each collector reports a set of metrics where each metric is identified by a key in a format that is specific for each collector.
Enable Streaming (enable_streaming)
Makes samples of output stages streamed if enabled. An optional boolean that defaults to False. If set to True, all output stages of this processor are streamed in the generic protobuf schema.