Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

 

AppFormix with Kafka

 

Set Up Kafka

Note

AppFormix does not explicitly create Kafka topics. The Kafka broker cluster should be configured to auto-create topics. Alternatively, you can manually manage the topic creation. If you already have Kafka running, you can skip Set Up Kafka and go directly to Set Up AppFormix with Kafka.

Setting up Kafka as a Docker container:

  1. Create a Docker network for this Kafka container and its dependencies to be connected to by running the following command:
  2. Next, bring up Zookeeper for Kafka to work:
  3. Bring up the Kafka container by running the following. The variable ip_address must be specified appropriately.
  4. If Kafka with SSL is required, then additional parameters are required:

    The secret_files_path should be replaced where all of the keystore and truststore files are present. For an example of how to create the above keystores and truststores, reference: https://github.com/confluentinc/cp-docker-images/blob/5.2.1-post/examples/kafka-cluster-ssl/secrets/create-certs.sh.

    Note

    If Kafka with SSL is required, then all of the hosts monitored by Appformix must have at least Python version of 2.7.9. It is also required that the Certificate Authority (CA) used for the certificates for the Kafka broker(s) be a trusted CA on all of the hosts monitored by AppFormix. In order for AppFormix containers to communicate with the Kafka broker(s), the CA file must be set as a group_vars/all variable appformix_kafka_ssl_ca at installation time.

    Now that Kafka is set up, next you can configure AppFormix with Kafka.

Set Up AppFormix with Kafka

To configure AppFormix with Kafka, a POST request must be sent to an AppFormix Controller API:

The following fields must be sent in this request:

NameThe name of the Kafka cluster, which can be anything.
BootstrapServers A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. Each item in the list is a string in the format host:port.

To send a POST body request using Ansible:

  1. Run the POST body request, which is similar to the following:
  2. If Kafka has been set up with SSL, then an additional field is needed:

To send a POST body request from the AppFormix Dashboard:

  1. Select Settings in the upper right corner, then select AppFormix Settings > Kafka. Next, click + Add Config.
    Figure 1: AppFormix Settings for Kafka Page
    AppFormix Settings for Kafka Page
  2. Enter a name for the Kafka configuration and list the BootstrapServers as a comma separated list of strings with each string in the host:port format.
    Figure 2: Enter Information for Kafka Configuration Name and Bootstrap Servers
    Enter Information for Kafka Configuration Name and
Bootstrap Servers
  3. Click Setup after the fields have been populated. You will then see the Kafka configuration as shown in Figure 3.
    Figure 3: Completed AppFormix Settings for Kafka Configuration
    Completed AppFormix Settings
for Kafka Configuration
    Note

    The following steps are for streaming network telemetry data to Kafka. All AppFormix alarms are automatically sent to Kafka once Kafka has been configured as stated in the earlier procedures above. There are no additional steps needed for alarms. See AppFormix Alarms With Kafka.

  4. Click + Add Subscription to create a subscription.
    Figure 4: Add Subscription for Streaming Network Telemetry Data to Kafka
    Add Subscription for Streaming Network Telemetry Data
to Kafka
  5. Next create a Topic, select devices, and then select which Sensors/MIBs you want sent to Kafka. The specified data will then be sent to Kafka under the specified topic. Click Create Subscription after the fields are populated.
    Figure 5: Create Topic, Select Devices and Sensors/MIBs to Send to Kafka
    Create Topic, Select Devices and Sensors/MIBs to Send
to Kafka

    You can see all of your successfully added Subscriptions in the table.

    Figure 6: Successfully Added Subscription for Streaming Network Telemetry Data to Kafka
    Successfully Added Subscription for Streaming Network
Telemetry Data to Kafka

Messages from AppFormix to Kafka

After configuration, messages from AppFormix are received by an appropriate Kafka consumer.

In the following command, bootstrap_server is one of the bootstrap servers specified in the BootstrapServers variable above and topic is the topic that was specified in the subscription created. :

This command outputs messages to standard output. Output for topic grpc-components with sensor /components/ selected will look something like:

AppFormix Alarms With Kafka

AppFormix Alarms are configured to automatically send alerts to Kafka, if Kafka has been configured in AppFormix. See Set Up AppFormix with Kafka. AppFormix sends alarms with the topic as the alarm's name. For example, Alarm name host_cpu is sent to Kafka with topic host_cpu.