Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

 

Installing Data Collectors for Analytics

 

The Analytics functionality streams data from the network devices, via data collectors, to the NorthStar Controller where it is processed, stored, and made available for viewing in the web UI. You can install data collectors either in the same machine as the NorthStar Controller application server (single-server deployment) or in other machines that are dedicated to log collection and storage (standalone deployment). In both cases, the supplied install scripts take care of installing the required packages and dependencies.

Single Server Deployment

To install the data collector together with the application server, use the following procedure:

  1. Install the NorthStar Controller bundle as usual, using the install.sh script. See the NorthStar Controller Getting Started Guide.
  2. Run the install-analytics.sh script.
    [root@ns ~]# cd /opt/northstar/northstar_bundle_3.0.0/
    [root@ns northstar_bundle_3.0.0]# ./install-analytics.sh

The configuration options from the collector processes are read from the /opt/northstar/data/northstar.cfg file. In a single server deployment, no special changes are required because the parameters needed to start up the collector are part of the default configuration. For your reference, Table 1 lists some of the settings that the collector processes read from the file.

Table 1: Some of the Settings Read by Collector Processes

Setting

Description

mq_host

Points to the IP address or VIP (for multiple application server deployments) of hosts running the messaging bus service (the NorthStar application server). Defaults to localhost if not present.

mq_username

Username used to connect to the messaging bus. Defaults to northstar.

mq_password_enc

Password used to connect to the messaging bus. There is no default; the service fails to start if this is not configured. On single-server deployments, the password is set during the normal application install process.

mq_port

TCP port number used by the messaging bus. Defaults to 5672.

es_port

TCP port used by elasticsearch. Defaults to 9200.

es_cluster_name

Used by elasticsearch in HA scenarios to form a cluster. Nodes in the same cluster must be configured with the same cluster name. Defaults to NorthStar.

jvision_ifd_port, jvision_ifl_port and jvision_lsp_port

UDP port numbers the collector listens to for telemetry packets from the devices. Default to 2000, 2001 and 2002, respectively.

rpmstats_port

Used to read syslog messages generated from the device with the results of the RPM stats. Defaults to 1514.

Standalone Deployment

Figure 1 shows a sample configuration with a single application server node and three collectors. They all connect to the same Ethernet network, through the eth1 interface in all nodes. The following procedure uses this configuration to demonstrate the standalone deployment option.

Figure 1: Example Standalone Deployment with Three Collectors
Example Standalone
Deployment with Three Collectors

To install a standalone data collector, use the following procedure.

  1. Install the NorthStar Controller bundle as usual, using the install.sh script. See the NorthStar Controller Getting Started Guide. If you are configuring an HA cluster, install the bundle on each HA cluster member.
  2. On the data collection machine, install northstar_bundle.rpm, but do not run the install.sh script. Instead, run the install-analytics.sh script. The script installs all required dependencies such as NorthStar-JDK, NorthStar-Python, and so on.​
    [root@ns2 northstar_bundle_3.0.0]# /opt/northstar/northstar_bundle_3.0.0/install-analytics.sh
  3. The next configuration steps require you to run the net_setup.py script to configure the application server and the collectors so they can connect to each other. But before you do that, we recommend that you copy the public SSH key of the server where the net_setup.py script is to be executed to all other nodes. The net_setup.py script can be run on either the application server or one of the data collectors to configure all the machines. This is not a required step, but it saves typing the passwords of all the systems later when the script is deploying the configurations or testing the connectivity to the different nodes.
    [root@ns network-scripts]# ssh-copy-id root@192.168.10.200

    Try logging into the machine using ssh root@192.168.10.200 and check in with .ssh/authorized_keys.

    Repeat this process for all nodes (192.168.10.100, 192.168.10.200, 192.168.10.201, and 192.168.10.202 in our example).

  4. Run net_setup.py on the application server or one of the collectors. The Main Menu is displayed:
  5. Select G.) Data Collector Setting. The Data Collector Configuration Settings menu is displayed.
  6. Select options from the Data Collector Configuration Settings menu to make the following configuration changes:
    • Select 3 to modify the application server settings, and configure the application server name and IP address. For example:

      Please select a number to modify.
      [CR=return to main menu]:
      3
    • Select 4 to modify the collector IP address. For example:

      Please select a number to modify.
      [CR=return to main menu]:
      4
    • Select 2 to add additional collectors as needed. In our example, two additional collectors would be added. For example:

      Please select a number to modify.
      [CR=return to main menu]:
      2
      Please select a number to modify.
      [CR=return to main menu]:
      2
    • Select 8A to configure a virtual IP (VIP) address for the cluster of collector nodes. For example:

      Please select a number to modify.
      [CR=return to main menu]:
      8A

      This VIP serves two purposes:

      • It allows the application server to send queries to a single endpoint. The VIP will be active on one of the nodes, and will switch over in the event of a failure (a full node failure or failure of any of the processes running in the collectors).

      • Devices can send telemetry data to the VIP, ensuring that if a collector fails, the telemetry data can still be processed by whichever non-failing node takes ownership of the VIP.

    The configuration for our example should now look like this:

  7. Select 9 to test connectivity between nodes. For example:
    Please select a number to modify.
    [CR=return to main menu]:
    8
  8. Select A or B to configure all nodes for the deployment. Note

    This option restarts the nodejs process in the NorthStar application node.

    For our example, select B:

    Please select a number to modify.
    [CR=return to main menu]:
    B
    YES

    This completes the installation and telemetry data can now be sent to the collectors using the collector VIP or any of the individual node IP addresses.

  9. Verify that data collection is working by checking that all services are running.
    [root@collector01 northstar_bundle_3.0.0]# supervisorctl status

    The collectors should start processing all records they get from the network, and pushing statistics to the application server through rabbitmq. Check the pcs.log to see the statistics being pushed to the PC server. For example:

    You can also use the REST APIs to get some aggregated statistics. This tests the path from client to nodejs to elasticsearch.

    The following logs are available to help with troubleshooting:

    • /opt/northstar/logs/elasticsearch.msg

    • /opt/northstar/logs/logstash.msh

    • /opt/northstar/logs/logstash.log