Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

Navigation
Guide That Contains This Content
[+] Expand All
[-] Collapse All

    Log and Flow Information APIs

    In Contrail, log and flow analytics information is collected and stored using a horizontally scalable Contrail collector and NoSQL database. The analytics-api server provides REST APIs to extract this information using queries. The queries use well-known SQL syntax, hiding the underlying complexity of the NoSQL tables.

    HTTP GET APIs

    Use the following GET APIs to identify tables and APIs available for querying.

    /analytics/tables -- lists the SQL-type tables available for querying, including the hrefs for each of the tables

    /analytics/table/<table> -- lists the APIs available to get information for a given table

    /analytics/table/<table>/schema -- lists the schema for a given table

    HTTP POST API

    Use the following POST API information to extract data from a table.

    /analytics/query -- format your query using the following SQL syntax:

    • SELECT field1, field2 ...
    • FROM table1
    • WHERE field1 = value1 AND field2 = value2 ...
    • FILTER BY ...
    • SORT BY ...
    • LIMIT n

    Additionally, it is mandatory to include the start time and the end time for the data range to define the time period for the query data. The parameters of the query are passed through POST data, using the following fields:

    • start_time — the start of the time period
    • end_time — the end of the time period
    • table — the table from which to extract data
    • select_fields — the columns to display in the extracted data
    • where — the list of match conditions

    POST Data Format Example

    The POST data is in JSON format, stored in an idl file. A sample file is displayed in the following.

    Note: The result of the query API is also in JSON format.

    /*
    * Copyright (c) 2013 Juniper Networks, Inc. All rights reserved.
    */
    
    /*
    *  query_rest.idl
    *
    *  IDL definitions for query engine REST API
    *
    *  PLEASE NOTE: After updating this file, do update json_parse.h
    *
    */
    
    enum match_op {
        EQUAL = 1,
        NOT_EQUAL = 2,
        IN_RANGE = 3,
        NOT_IN_RANGE = 4,   // not supported currently
        // following are only for numerical column fields
        LEQ = 5, // column value is less than or equal to filter value
        GEQ = 6, // column value is greater than or equal to filter value
        PREFIX = 7, // column value has the "value" field as prefix
        REGEX_MATCH = 8 // for filters only
    }
    
    enum sort_op {
        ASCENDING = 1,
        DESCENDING = 2,
    }     
        
    struct match {
        1: string name;
        2: string value;
        3: match_op op;
        4: optional string value2;   // this is for only RANGE match
    }
    
    typedef list<match> term; (AND of match)  
    
    enum flow_dir_t {
         EGRESS = 0,
         INGRESS = 1 
    }
    struct query {
        1: string table; // Table to query (FlowSeriesTable, MessageTable, ObjectVNTable, ObjectVMTable, FlowRecordTable)
        2: i64 start_time; // Microseconds in UTC since Epoch
        3: i64 end_time; // Microseconds in UTC since Epoch
        4: list<string>> select_fields; // List of SELECT fields
        5: list<term> where; // WHERE (OR of terms)
        6: optional sort_op sort;
        7: optional list<string> sort_fields; 
        8: optional i32 limit;
        9: optional flow_dir_t dir; // direction of flows being queried
        10: optional list<match> filter; // filter the processed result by value
    }
    
    struct flow_series_result_entry {
        1: optional i64 T;  // Timestamp of the flow record
        2: optional string sourcevn;
        3: optional string sourceip;
        4: optional string destvn;
        5: optional string destip;
        6: optional i32 protocol;
        7: optional i32 sport;
        8: optional i32 dport;
        9: optional flow_dir_t direction_ing;
        10: optional i64 packets;  // mutually exclusive to 12,13
        11: optional i64 bytes; // mutually exclusive to 12,13
        12: optional i64 sum_packets; // represented as "sum(packets)" in JSON
        13: optional i64 sum_bytes; // represented as "sum(bytes)" in JSON
    };
    typedef list<flow_series_result_entry> flow_series_result;

    Query Types

    The analytics-api supports two types of queries. Both types use the same POST parameters as described in POST API.

    • sync — Default query mode. The results are sent inline with the query processing.
    • async — To execute a query in async mode, attach the following header to the POST request: Expect: 202-accepted.

    Examining Query Status

    For an asynchronous query, the analytics-api responds with the code: 202 Accepted. The response contents are a status entity href URL of the form: /analytics/query/<QueryID>. The QueryID is assigned by the analytics-api. To view the response contents, poll the status entity by performing a GET action on the URL. The status entity has a variable named progress, with a number between 0 and 100, representing the approximate percentage completion of the query. When progress is 100, the query processing is complete.

    Examining Query Chunks

    The status entity has an element named chunks that lists portions (chunks) of query results. Each element of this list has three fields: start_time, end_time, href. The analytics-api determines how many chunks to list to represent the query data. A chunk can include an empty string ("") to indicate that the data query is not yet available. If a partial result is available, the chunk href is of the form: /analytics/query/<QueryID>/chunk-partial/<chunk number>. When the final result of a chunk is available, the href is of the form: /analytics/query/<QueryID>/chunk-final/<chunk number>.

    Example Queries for Log and Flow Data

    The following example query lists the tables available for query.

    [root@host ~]# curl 127.0.0.1:8081/analytics/tables | python -mjson.tool
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100   846  100   846    0     0   509k      0 --:--:-- --:--:-- --:--:--  826k
    [
        {
            "href": "http://127.0.0.1:8081/analytics/table/MessageTable",
            "name": "MessageTable"
        },
        {
            "href": "http://127.0.0.1:8081/analytics/table/ObjectVNTable",
            "name": "ObjectVNTable"
        },
        {
            "href": "http://127.0.0.1:8081/analytics/table/ObjectVMTable",
            "name": "ObjectVMTable"
        },
        {
            "href": "http://127.0.0.1:8081/analytics/table/ObjectVRouter",
            "name": "ObjectVRouter"
        },
        {
            "href": "http://127.0.0.1:8081/analytics/table/ObjectBgpPeer",
            "name": "ObjectBgpPeer"
        },
        {
            "href": "http://127.0.0.1:8081/analytics/table/ObjectRoutingInstance",
            "name": "ObjectRoutingInstance"
        },
        {
            "href": "http://127.0.0.1:8081/analytics/table/ObjectXmppConnection",
            "name": "ObjectXmppConnection"
        },
        {
            "href": "http://127.0.0.1:8081/analytics/table/FlowRecordTable",
            "name": "FlowRecordTable"
        },
        {
            "href": "http://127.0.0.1:8081/analytics/table/FlowSeriesTable",
            "name": "FlowSeriesTable"
        }
    ]
    

    The following example query lists details for the table named MessageTable.

    [root@host ~]# curl 127.0.0.1:8081/analytics/table/MessageTable | python -mjson.tool
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100   192  100   192    0     0   102k      0 --:--:-- --:--:-- --:--:--  187k
    [
        {
            "href": "http://127.0.0.1:8081/analytics/table/MessageTable/schema",
            "name": "schema"
        },
        {
            "href": "http://127.0.0.1:8081/analytics/table/MessageTable/column-values",
            "name": "column-values"
        }
    ]

    The following example query lists the schema for the table named MessageTable.

    [root@host ~]# curl 127.0.0.1:8081/analytics/table/MessageTable/schema | python -mjson.tool
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100   630  100   630    0     0   275k      0 --:--:-- --:--:-- --:--:--  307k
    {
        "columns": [
            {
                "datatype": "int",
                "index": "False",
                "name": "MessageTS"
            },
            {
                "datatype": "string",
                "index": "True",
                "name": "Source"
            },
            {
                "datatype": "string",
                "index": "True",
                "name": "ModuleId"
            },
            {
                "datatype": "string",
                "index": "True",
                "name": "Category"
            },
            {
                "datatype": "int",
                "index": "True",
                "name": "Level"
            },
            {
                "datatype": "int",
                "index": "False",
                "name": "Type"
            },
            {
                "datatype": "string",
                "index": "True",
                "name": "Messagetype"
            },
            {
                "datatype": "int",
                "index": "False",
                "name": "SequenceNum"
            },
            {
                "datatype": "string",
                "index": "False",
                "name": "Context"
            },
            {
                "datatype": "string",
                "index": "False",
                "name": "Xmlmessage"
            }
        ],
        "type": "LOG"
    }
    
    

    The following set of example queries explore a message table.

    root@a6s45:~# cat filename
    { "end_time": "now" , "select_fields": ["MessageTS", "Source", "ModuleId", "Category", "Messagetype", "SequenceNum", "Xmlmessage", "Type", "Level", "NodeType", "InstanceId"] , "sort": 1 , "sort_fields": ["MessageTS"] , "start_time": "now-10m" , "table": "MessageTable" , "where": {"name": "ModuleId", "value": "contrail-control", "op": 1, "suffix": null, "value2": null}, {"name": "Messagetype", "value": "BGPRouterInfo", "op": 1, "suffix": null, "value2": null} }
    
    root@a6s45:~#
    root@a6s45:~# curl -X POST --data @filename 127.0.0.1:8081/analytics/query --header "Content-Type:application/json" | python -mjson.tool
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100  9765    0  9297  100   468   9168    461  0:00:01  0:00:01 --:--:--  9177
    {
        "value": [
            {
                "Category": null,
                "InstanceId": "0",
                "Level": 2147483647,
                "MessageTS": 1428442589947392,
                "Messagetype": "BGPRouterInfo",
                "ModuleId": "contrail-control",
                "NodeType": "Control",
                "SequenceNum": 1302,
                "Source": "a6s45",
                "Type": 6,
                "Xmlmessage": "<BGPRouterInfo type=""><data type=""><BgpRouterState><name type=""
    >a6s45</name><cpu_info type=""><CpuLoadInfo><num_cpu type="">4</num_cpu
    ><meminfo type=""><MemInfo><virt type="">438436</virt><peakvirt type=""
    >561048</peakvirt><res type="">12016</res></MemInfo></meminfo><cpu_share
    type="">0.0416667</cpu_share></CpuLoadInfo></cpu_info><cpu_share type=""
    >0.0416667</cpu_share></BgpRouterState></data></BGPRouterInfo>"        },
            {
                "Category": null,
                "InstanceId": "0",
                "Level": 2147483647,
    
    ...

    Modified: 2015-09-02