Configuring the Streaming API

What Is Kafka?

Kafka is an event-streaming platform that allows real-time capture of data sent from various event sources (sensors, databases, mobile devices) in the form of event streams, as well as durable storing of these event streams for later retrieval and manipulation.

With Kafka it is possible to manage the event streaming end-to-end in a distributed, highly scalable, elastic, fault-tolerant, and secure manner.

Note:

Kafka can be configured in many different ways and was designed for scalability and redundant systems. This document focuses only on how to configure it to make use of the Streaming API feature found in Paragon Active Assurance Control Center. For more advanced setups we refer to the official Kafka documentation: kafka.apache.org/26/documentation.html.

Terminology

• Kafka: Event-streaming platform.
• Kafka topic: Collection of events.
• Kafka subscriber/consumer: Component responsible for retrieval of events stored in a Kafka topic.
• Kafka broker: Storage layer server of a Kafka cluster.
• SSL/TLS: SSL is a secure protocol developed for sending information securely over the Internet. TLS is the successor of SSL, introduced in 1999.
• SASL: Framework that provides mechanisms for user authentication, data integrity checking, and encryption.
• Streaming API subscriber: Component responsible for retrieval of events stored in topics defined in Paragon Active Assurance and meant for external access.
• Certificate Authority: A trusted entity that issues and revokes public key certificates.
• Certificate Authority root certificate: Public key certificate that identifies a Certificate Authority.

How the Streaming API Works

As previously mentioned, the Streaming API allows external clients to retrieve information about metrics from Kafka.

All metrics collected by the Test Agents during a test or monitoring task are sent to the Stream service. After a processing phase, the Stream service publishes those metrics on Kafka together with additional metadata.

Enabling the Streaming API

Note:

These instructions are to be run on the Control Center server using sudo.

Since the Streaming API adds some overhead to the Control Center, it is not enabled by default. To enable the API, we must first enable publishing of metrics to Kafka in the main configuration file:

• /etc/netrounds/netrounds.conf

Warning:

Enabling this feature might impact Control Center performance. Make sure you have dimensioned your instance accordingly.

Next, to enable forwarding of these metrics to the correct Kafka topics:

• /etc/netrounds/metrics.yaml

To enable and start the Streaming API services, run:

Finally, restart the services:

Verifying That the Streaming API Works in Control Center

Note:

These instructions are to be run on the Control Center server.

You can now verify that you are receiving metrics on the correct Kafka topics. To do so, install the kafkacat utility:

If you have a test or monitor running in Control Center, you should be able to use kafkacat to receive metrics and metadata on these topics.

Replace myaccount with the short name of your account (this is what you see in your Control Center URL):

You should now see metrics by running this command:

To view metadata, run the following command (note that this will not update as frequently):

Note: This is just a sanity check to make sure things are being published correctly. The data you see being published will be in binary form, which kafkacat will not decode by default. For properly subscribing to these topics, please see the Client Examples section.

This verifies that we have a working Streaming API from within Control Center. However, most likely you are interested in accessing the data from an external client instead. The next section describes how to open up Kafka for external access.

Connecting to Kafka: Caveats

CAUTION:

Please read this carefully, since it is easy to run into connection issues with Kafka if you have not understood these concepts.

In the Control Center setup described in this document, there is only a single Kafka broker.

However, note that a Kafka broker is meant to run as part of a Kafka cluster which may consist of many Kafka brokers.

When connecting to a Kafka broker, an initial connection is set up by the Kafka client. Over this connection the Kafka broker in turn will return a list of "advertised listeners", which is a list of one or more Kafka brokers.

On receiving this list, the Kafka client will disconnect, then reconnect to one of these advertised listeners. The advertised listeners must contain hostnames or IP addresses that are accessible to the Kafka client, or the client will fail to connect.

If SSL encryption is used, involving an SSL certificate which is tied to a particular hostname, it is even more important that the Kafka client receives the correct address to connect to, since otherwise the connection may be rejected.

Read more about Kafka listeners here: www.confluent.io/blog/kafka-listeners-explained

SSL/TLS Encryption

To make sure only trusted clients are allowed to access Kafka and the Streaming API, we must configure the following:

• Authentication: Clients must provide username and password through an SSL/TLS secure connection between the client and Kafka.
• Authorization: Authenticated clients can perform tasks regulated by ACLs.

Here is an overview:

To fully understand how the SSL/TLS encryption works for Kafka, please refer to the official documentation: docs.confluent.io/platform/current/kafka/encryption.html

SSL/TLS Certificate Overview

Note:

In this subsection we will use the following terminology:

Certificate: An SSL certificate signed by a Certificate Authority (CA). Each Kafka broker has one.

Keystore: The keystore file that stores the certificate. The keystore file contains the private key of the certificate; therefore, it needs to be kept safely.

Truststore: A file containing the trusted CA certificates.

To set up the authentication between an external client and Kafka running in Control Center, both sides must have a keystore defined with a related certificate signed by a Certificate Authority (CA) together with the CA root certificate.

In addition to this, the client must also have a truststore with the CA root certificate.

The CA root certificate is common to the Kafka broker and Kafka client.

Creating the Required Certificates

This is covered in the Appendix.

Kafka Broker SSL/TLS Configuration in Control Center

Note:

These instructions are to be run on the Control Center server.

Note:

Before continuing, you must create the keystore which contains the SSL certificate by following the instructions in the Appendix. The paths mentioned below come from these instructions.

The SSL keystore is a file stored on disk with the file extension .jks.

Once you have the required certificates created for both the Kafka broker and the Kafka client available, you can continue by configuring the Kafka broker running in Control Center. You need to know the following:

• <public_hostname>: The public hostname of Control Center; this must be resolvable and accessible by Kafka clients.
• <keystore_pwd>: The keystore password provided when creating the SSL certificate.
• <admin_pwd> and <client_pwd>: These are the passwords you want to set for the admin and client user respectively. Note that you can add more users, as indicated in the example.

Edit or append (with sudo access) the properties below in /etc/kafka/server.properties, inserting the above variables as shown:

Warning:

Do not remove PLAINTEXT://localhost:9092; this will break Control Center functionality since internal services will not be able to communicate.

Setting up Access Control Lists (ACLs)

Turning On ACLs on localhost

Warning:

We must first set up ACLs for localhost, so that Control Center itself can still access Kafka. If this is not done, things will break.

We then need to enable ACLs for external read-only access, so that external users are allowed to read the paa.public.* topics.

Note:

For more fine-grained control, please refer to the official Kafka documentation.

Once done with this, you need to restart the services:

To verify that a client can establish a secure connection, run the following command on an external client computer (not on the Control Center server). Below, PUBLIC_HOSTNAME is the Control Center hostname:

To ensure that internal services have been granted access to the Kafka server, please check the following logfiles:

• /var/log/kafka/server.log
• /var/log/kafka/kafka-authorizer.log

kafkacat

Note:

These instructions are to be run on a client computer (not on the Control Center server).

Note:

To display metrics information, ensure that at least one monitor is running in Control Center.

To verify and validate connectivity as an external client, it is possible to use the kafkacat utility which was installed in the section Verifying That the Streaming API Works in Control CenterVerifying That the Streaming API Works in Control Center.

Perform the following steps:

Note:

Below, CLIENT_USER is the user previously specified in the file /etc/kafka/server.properties in Control Center: namely, user_client and the password set there.

The CA root certificate used to sign the server side SSL certificate must be present on the client.

• Create a file client.properties with the following content:

  where

  • {PATH_TO_CA_CERT} is the location of the CA root certificate used by the Kafka broker
  • {CLIENT_USER} and {CLIENT_PASSWORD} are the user credentials for the client.

• Run the following command to see the message consumed by kafkacat:

  where {METRICS_TOPIC} is the name of the Kafka topic with prefix "paa.public.".

Note:

Older versions of kafkacat do not provide the -F option for reading the client settings from a file. If you are using such a version, you must provide the same settings from the command line as shown below.

To debug the connectivity, you can use the -d option:

Be sure to refer to the documentation for the Kafka client library in use, as the properties may differ from those in client.properties.

Metrics Protobuf Schema

Metadata Protobuf Schema

Installing and Configuring Client Examples

You find client-examples in the Paragon Active Assurance Control Center folder:

To install client-examples on your external client computer, proceed as follows:

client-examples requires Docker to run. Downloads and installation instructions for Docker can be found at https://docs.docker.com/engine/install.

Using Client Examples

The client-examples tools can run in either basic or advanced mode to build examples of varying complexity. In both cases, it is also possible to run the examples with a configuration file containing additional properties for further customization of the client side.

Basic Mode

In basic mode, the metrics and their metadata are streamed separately. To this end, the client listens to each Kafka topic available for external access and simply prints the received messages to the console.

To start execution of the basic examples, run:

where ACCOUNT_SHORTNAME is the short name of the account you want to get the metrics from.

To terminate the execution of the example, press Ctrl + C. (There may be a slight delay before the execution stops because the client waits for a timeout event.)

Advanced Mode

Note:

Metrics are displayed only for HTTP monitors running in Control Center.

Execution in advanced mode shows the correlation between metrics and metadata messages. This is possible thanks to the presence in each metrics message of a stream id field which refers to the corresponding metadata message.

To execute the advanced examples, run:

where ACCOUNT_SHORTNAME is the short name of the account you want to get the metrics from.

To terminate the execution of the example, press Ctrl + C. (There may be a slight delay before the execution stops because the client waits for a timeout event.)

Additional Settings

It is possible to run the examples with additional configuration of the client using the --config-file option followed by a file name containing properties in the form key=value.

Note:

All files referenced in the command above must be located in the current directory and referred using only relative paths. This applies both to the --config-file argument and to all entries in the configuration file that describe file locations.

Validating External Client Authentication

To validate client authentication from outside the Control Center using client-examples, perform the following steps:

• From the Paragon Active Assurance Control Center folder, switch to the paa-streaming-api-client-examples folder:

• Copy the CA root certificate ca-cert into the current directory.

• Create a client.properties file with the following content:

  where {CLIENT_USER} and {CLIENT_PASSWORD} are the user credentials for the client.

• Run basic examples:

  where ACCOUNT_SHORTNAME is the short name of the account you want to get the metrics from.

• Run advanced examples:

Creating a Kafka Broker Certificate

Creating a Certificate Using a Real Certificate Authority (Recommended)

It is recommended that you get a real SSL certificate from a trusted CA.

Once you have decided on a CA, copy their CA root certificate ca-cert file to your own path as shown below:

Create Your Own Certificate Authority

Note:

Normally you should have your certificate signed by a real Certificate Authority; see the preceding subsection. What follows is just an example.

Here we create our own Certificate Authority (CA) root certificate file valid for 999 days (not recommended in production):

Creating the Client Truststore

Now you can create a truststore file that contains the ca-cert generated above. This file will be needed by the Kafka client that will access the Streaming API:

Now that the CA certificate is in the truststore, the client will trust any certificate signed with it.

You should copy the file kafka.client.truststore.jks to a known location on your client computer and point to it in the settings.

Creating the Keystore for the Kafka Broker

To generate the Kafka broker SSL certificate and then the keystore kafka.server.keystore.jks, proceed as follows:

Generating the SSL Certificate

Below, 999 is the number of days of validity of the keystore, and FQDN is the fully qualified domain name of the client (public host name of the node).

Note:

It is important that the FQDN matches the exact hostname that the Kafka client will use to connect to the Control Center.

Create a certificate signing request and store it in the file named cert-server-request:

You should now send the file cert-server-request to your Certificate Authority (CA) if you are using a real one. They will then return the signed certificate. We will refer to this as cert-server-signed below.

Signing the SSL Certificate Using a Self-created CA Certificate

Note:

Again, using your own CA is not recommended in a production system.

Sign the certificate using the CA by means of the file cert-server-request, which produces the signed certificate cert-server-signed. See below; ca-password is the password set when creating the CA certificate.

Importing the Signed Certificate into the Keystore

Import the ca-cert root certificate into the keystore:

Import the signed certificate referred to as cert-server-signed:

The file kafka.server.keystore.jks should be copied to a known location on the Control Center server, and then referred to in /etc/kafka/server.properties.