Build and Load BYOI Custom Plug-in Images
SUMMARY
To send data to Paragon Automation using Bring Your Own Ingest (BYOI) Custom Plug-ins, you must code the input plug-in and create a plug-in ingest image file with a shell script and a process file. Paragon Automation executes the shell script when configurations of the BYOI ingest and device group (mapped to the ingest) change. The ingest image also contains a Kubernetes YAML file for the ingest container. The Kubernetes YAML file contains configurations that enable the Kubernetes engine to start and to stop the service for a BYOI plug-in ingest.
The workflow to build and to load the BYOI custom plug-in image is as follows:
-
Create a process file to write data to the Paragon Automation database and a shell script for configuration updates.
- See Create a Process File for the Plug-in Image for an example Python script (process file) that parses attributes in JSON configuration file to send data to the Paragon Insights database.
- See Create a Shell Script for Configuration Updates for an example shell script.
-
Tag the image file and export the image as a compressed tar file. See Tag and Export the BYOI Custom Plug-in Image for commands to tag and export the image file.
-
Modify the Kubernetes Jinja template to create a YAML file for the Kubernetes container pod. The container pod is where the BYOI ingest is deployed in Paragon Automation. See Configure Kubernetes YAML File for a sample Kubernetes Jinja template file.
-
(Optional) Assign a different IP address if you want the BYOI plug-in to be accessible for external applications. See Assign Virtual IP Address to Plug-in for a Kubernetes Jinja template in which you can assign a custom virtual IP address for the BYOI plug-in.
-
Load the compressed tar file and Kubernetes YAML file to the Paragon Automation primary node. See Load the BYOI Custom Plug-in to load the BYOI plug-in image file and the Kubernetes YAML file.
Create a Process File for the Plug-in Image
You can create a process file, such as the following example Python file, and include it in the BYOI plug-in image. When you run the image in a Kubernetes container, Paragon Automation executes the process file. The following sample Python file uses the attributes that are described in Table 1 to send a key (a random integer between 0 and 9 in the example file) for the measurement (topic/rule/sensor_name/byoi) to the database.
import requests
import time
import random
import os
import json
# read tand_host, tand_port from env vars
tand_host = os.environ.get('TAND_HOST', 'localhost')
tand_port = os.environ.get('TAND_PORT', '3000')
# read device, plugin, rule related attributes from config json
with open('/etc/byoi/config.json', 'r') as f:
config_json = json.load(f)
# input, device, sensor as lists. modify index as needed. Using 0 for all idxes
database = config_json['hbin']['inputs'][0]['plugin']['config']['device'][0]['healthbot-storage']['database']
measurement = config_json['hbin']['inputs'][0]['plugin']['config']['device'][0]['sensor'][0]['measurement']
# Construct post request and data
url = 'http://{}:{}/write?db={}'. \
format(tand_host, tand_port, database)
data = '{} {} {}'
metric = 'key'
while True:
fields = '{}={}'.format(metric, random.randint(0,9))
timestamp = int(time.time()) * 1000000000
x = requests.post(url, data=data.format(measurement, fields, timestamp))
time.sleep(10)In the Python example file, use the following URL format to send data to Paragon Automation.
url = 'http://{}:{}/write?db={}'.
\format(tand_host, tand_port, database)The value for the database attribute must follow the syntax database-name:device-group-name:device-name.
The line protocol (post body) contains a string in the following format.
data = '{} {} {}'.format(measurement, fields,
timestamp)When you create an instance of a custom BYOI plug-in, a JavaScript Object Notation (JSON) configuration file is attached as a volume in the Kubernetes container for the BYOI ingest instance. The JSON configuration file contains information such as the device, device group, sensor path, hostname, and port of the backend service to which ingest data is sent. You can go through the JSON configuration using /etc/byoi/config.json file for all the available attributes.
Table 1 lists several key attributes in the JSON configuration file.
| Attributes | Description | How to Access the Attributes |
|---|---|---|
| tand_host |
Host name of the back-end service to which the plug-in sends the ingest data. |
Environment Variable $TAND_HOST |
| tand_port |
Port number of the back-end service to which the plug-in sends data. |
Environment Variable $TAND_PORT |
| database |
Name of database that stores the ingest data. The value for this attribute differs for each Paragon Automation node. |
config_json['hbin'] ['inputs']['plugin']['config'] ['device'][idx]['healthbot-storage'] ['database'] |
| measurement |
Measurement of database in line protocol. For example, topic/rule/sensor_name/byoi Note:
The value of sensor_name differs from sensor to sensor. See the https://docs.influxdata.com/influxdb to learn more about measurements. |
config_json['hbin']['inputs']['plugin'] ['config']['device'][idx]['sensor'] [sensor_idx]['measurement'] |
| fields |
Metric-value pairs, separated by comma without space. For example, cpu_usage=50,memory_utilization=12. |
None |
| timestamp | Unix Epoch timestamp in nanoseconds. | None |
| password |
Encoded password of the device that receives the streaming data. See Decode Device Passwordto decode the device password. |
config_json['hbin'] ['inputs']['plugin']['config'] ['device'][idx]['authentication'] ['password']['password'] |
Decode Device Password
The JSON configuration file can contain encoded sensitive information such as the password of the device that streams data.
To decode the data, you can
initiate a POST call using the API
api-server:9000/api/v2/junos-decode inside the plug-in
container, with the encoded data in the post body.
The following sample POST call decodes an encoded password present in the JSON configuration file.
curl -X POST -L api-server:9000/api/v2/junos-decode -H
"Content-Type: application/json" -d '{"data": "$ABC123"}' -vCreate a Shell Script for Configuration Updates
When the BYOI ingest image configuration or the Paragon Automation device group configuration changes, the JSON configuration file is updated. When a change in configuration occurs, Paragon Automation signals BYOI about the configuration update by executing the shell script located at /jfit_scripts/jfit_reconfigure.sh. When you build the ingest plug-in image, you must name your shell script jfit_reconfigure.sh and copy the script to /jfit_scripts/ folder.
In the shell script, you can send a SIGHUP signal to the main process or simply kill old processes and start new ones. The following example shell script sends a SIGHUP signal to the main plug-in process:
pid=`ps -ef | grep ".*main.py" | grep -v 'grep' | awk '{ print $1}'` && \
kill -s HUP $pidTag and Export the BYOI Custom Plug-in Image
After you build the custom plug-in, you must tag the plug-in image and
export it as a tar file. You can tag the plug-in image in the
healthbot_plugin_name:your_version
format. You must export the plug-in image as a compressed tar file using the
following command:
docker save tag -o
healthbot_plugin_name.tar.gzConfigure Kubernetes YAML File
The Kubernetes Jinja template file has the basic configuration required to deploy Kubernetes resources such as containers for the for the BYOI ingest pod.
You can use the following sample Kubernetes Jinja template to create a YAML file. You must replace:
-
Placeholders for commands and args, <ADD_COMMAND> and <ADD_ARGUMENTS>.
For example, replace <ADD_COMMAND> with python3 and <ADD_ARGUMENTS> with the name of your Python file.
-
<PLUGIN_NAME_CAPITALIZED> with your plug-in name in upper case letters.
You can add other properties to the 'containers' part, such as volumes or Kubernetes secrets, in the template. After you modify the sample Kubernetes Jinja template, change the name of the file to healthbot_<plugin_name>.yml.j2 and save it.
The following code is a sample Kubernetes Jinja template.
set sg_name = '-' + env['SUBGROUP'] -%}
{%- set sg_dir = '_' + env['SUBGROUP'] -%}
{% if env['SUBGROUP'] == '' -%}
{%- set sg_name = '' -%}
{%- set sg_dir = '' -%}
{%- endif %}
kind: ConfigMap
apiVersion: v1
metadata:
namespace: {{ env['NAMESPACE'] }}
name: {{ env['GROUP_TYPE'] }}-{{ env['GROUP_NAME_VALID'] }}
{{ sg_name }}-{{ env['CUSTOM_PLUGIN_NAME'] }}
labels:
app: {{ env['CUSTOM_PLUGIN_NAME'] }}
group-name: {{ env['GROUP_NAME'] }}
group-type: {{ env['GROUP_TYPE'] }}
subgroup: {{ env['SUBGROUP'] }}
data:
TAND_HOST: '{{ env['GROUP_TYPE_SHORT'] }}-{{ env['GROUP_NAME_VALID'] }}
{{ sg_name }}-{{ env['CUSTOM_PLUGIN_NAME'] }}-terminus'
TAND_PORT: '{{env['tand:TAND_PORT']}}'
PUBLISHD_HOST: '{{env['publishd:PUBLISHD_HOST']}}'
PUBLISHD_PORT: '{{env['publishd:PUBLISHD_PORT']}}'
CONFIG_MANAGER_PORT: {{env['configmanager:CONFIG_MANAGER_PORT']}}
CHANNEL: '{{ env['GROUP_TYPE'] }}-{{ env['GROUP_NAME'] }}'
GODEBUG: 'madvdontneed=1'
IAM_SERVER: '{{ env['iam:IAM_SERVER'] }}'
IAM_SERVER_PORT: '{{ env['iam:IAM_SERVER_PORT'] }}'
IAM_SERVER_PROTOCOL: '{{ env['iam:IAM_SERVER_PROTOCOL'] }}'
IAM_NAMESPACE: '{{ env['iam:IAM_NAMESPACE'] }}'
---
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: {{ env['NAMESPACE'] }}
name: {{ env['GROUP_TYPE'] }}-{{ env['GROUP_NAME_VALID'] }}
{{ sg_name }}-{{ env['CUSTOM_PLUGIN_NAME'] }}
labels:
app: {{ env['CUSTOM_PLUGIN_NAME'] }}
group-name: {{ env['GROUP_NAME'] }}
group-type: {{ env['GROUP_TYPE'] }}
subgroup: {{ env['SUBGROUP'] }}
spec:
replicas: 1
selector:
matchLabels:
app: {{ env['CUSTOM_PLUGIN_NAME'] }}
group-name: {{ env['GROUP_NAME'] }}
group-type: {{ env['GROUP_TYPE'] }}
subgroup: {{ env['SUBGROUP'] }}
template:
metadata:
namespace: {{ env['NAMESPACE'] }}
labels:
app: {{ env['CUSTOM_PLUGIN_NAME'] }}
group-name: {{ env['GROUP_NAME'] }}
group-type: {{ env['GROUP_TYPE'] }}
subgroup: {{ env['SUBGROUP'] }}
spec:
tolerations:
- key: "node.kubernetes.io/not-ready"
operator: "Exists"
effect: "NoExecute"
tolerationSeconds: 180
- key: "node.kubernetes.io/unreachable"
operator: "Exists"
effect: "NoExecute"
tolerationSeconds: 180
initContainers:
- name: sync
image: {{ env['REGISTRY'] }}/{{ env['HEALTHBOT_INIT_CONTAINER_IMAGE'] }}:
{{ env['HEALTHBOT_INIT_CONTAINER_TAG'] }}
imagePullPolicy: Always
command: ["python3"]
args: ["/root/sync_files.py", "-c", "{{ env['GROUP_TYPE'] }}-
{{ env['GROUP_NAME'] }}"]
env:
- name: NODE_IP
valueFrom:
fieldRef:
fieldPath: status.hostIP
envFrom:
- configMapRef:
name: {{ env['GROUP_TYPE'] }}-{{ env['GROUP_NAME_VALID'] }}
{{ sg_name }}-{{ env['CUSTOM_PLUGIN_NAME'] }}
containers:
- name: {{ env['CUSTOM_PLUGIN_NAME'] }}
image: {{ env['REGISTRY'] }}/{{ env['HEALTHBOT_<<variable
>PLUGIN_NAME_CAPITALIZED</variable>>_IMAGE’] }}:
{{ env[’HEALTHBOT_<<variable>PLUGIN_NAME_CAPITALIZED</variable
>>_TAG’] }}
imagePullPolicy: Always
#example
#command: [“python3”]
#args: [“/main.py”]
command: [<<variable>ADD_COMMAND</variable>>]
args: [<<variable>ADD_ARGUMENTS</variable>>]
env:
- name: NODE_IP
valueFrom:
fieldRef:
fieldPath: status.hostIP
envFrom:
- configMapRef:
name: {{ env['GROUP_TYPE'] }}-{{ env['GROUP_NAME_VALID'] }}
{{ sg_name }}-{{ env['CUSTOM_PLUGIN_NAME'] }}
volumeMounts:
- name: default
mountPath: /etc/byoi
- name: data-model
mountPath: /etc/ml
volumes:
- name: default
hostPath:
type: DirectoryOrCreate
path: {{ env['JFIT_OUTPUT_PATH'] }}/{{ env['GROUP_NAME'] }}
{{ sg_dir }}/custom_{{ env['CUSTOM_PLUGIN_NAME'] }}_collector
- name: data-model
hostPath:
type: DirectoryOrCreate
path: {{ env['JFIT_ETC_PATH'] }}/data/models/{{ env['GROUP_NAME'] }}
imagePullSecrets:
- name: registry-secret
Assign Virtual IP Address to Plug-in
For a custom BYOI plug-in to be reachable from an external network, the plug-in needs to be exposed as a Kubernetes loadbalancer service. This is an optional configuration. By default, the plug-in uses virtual IP address of the Paragon Automation gateway. You can also assign a custom virtual IP address and add the following template to the end of the Kubernetes Jinja template file in Configure Kubernetes YAML File.
Ensure that you replace <PLUGIN_PORT> and <PROTOCOL> in the given template with your desired values such as port 80 for protocol HTTP. See the https://kubernetes.io/docs/concepts/services-networking/service/#protocol-support for supported protocols.
To configure a custom virtual IP address for the BYOI plug-in, replace <custom_load_balancer_ip> with a virtual IP address.
---
{% set service_values = env.get('SERVICE_VALUES', {}) -%}
{%- set global_annotations = service_values.get('annotations') -%}
{%- set global_load_balancer_ip = service_values.get('loadBalancerIP') -%}
{%- set custom_annotations = service_values.get(svc_name, {}).get('annotations') -%}
{%- set custom_load_balancer_ip = service_values.get(svc_name, {}).get('loadBalancerIP') -%}
{%- set service_type = service_values.get(svc_name, {}).get('type', 'LoadBalancer') -%}
{%- for ip in env['LOAD_BALANCER_IPS'] %}
apiVersion: v1
kind: Service
metadata:
namespace: {{ env['NAMESPACE'] }}
{%- if loop.index0 == 0 %}
name: {{ env['GROUP_TYPE_SHORT'] }}-{{ env['GROUP_NAME_VALID'] }}
{{ sg_name }}-{{ env['CUSTOM_PLUGIN_NAME'] }}
{%- else %}
name: {{ env['GROUP_TYPE_SHORT'] }}-{{ env['GROUP_NAME_VALID'] }}
{{ sg_name }}-{{ env['CUSTOM_PLUGIN_NAME'] }}-{{loop.index0}}
{%- endif %}
labels:
app: {{ env['CUSTOM_PLUGIN_NAME'] }}
group-name: {{ env['GROUP_NAME'] }}
group-type: {{ env['GROUP_TYPE'] }}
subgroup: '{{ env['SUBGROUP'] }}'
annotations:
{% if env.get('LOADBALANCER_PROVIDER', 'User') == 'HealthBot' %}
metallb.universe.tf/allow-shared-ip: healthbot-{{loop.index0}}
{% elif custom_annotations %}
{{ custom_annotations }}
{% elif global_annotations %}
{{ global_annotations }}
{% else %}
{}
{% endif %}
spec:
type: LoadBalancer
{%- if env.get('LOADBALANCER_PROVIDER', 'User') == 'HealthBot' %}
loadBalancerIP: {{ ip }}
{%- elif custom_load_balancer_ip %}
loadBalancerIP: {{ <custom_load_balancer_ip> }}
{%- elif global_load_balancer_ip %}
loadBalancerIP: {{ global_load_balancer_ip }}
{%- endif %}
selector:
app: {{ env['CUSTOM_PLUGIN_NAME'] }}
group-name: {{ env['GROUP_NAME'] }}
group-type: {{ env['GROUP_TYPE'] }}
subgroup: '{{ env['SUBGROUP'] }}'
ports:
- name: port
port: <PLUGIN_PORT>
protocol:<PROTOCOL>
{% endfor %}After you configure the port and protocol, external applications can communicate
with the custom BYOI plug-in via
<gateway_IP>:<PLUGIN_PORT>.
If you configured a custom virtual IP for the plug-in to act as a loadbalancer
service, external applications can communicate with the plug-in via
<custom_load_balancer_ip>:<PLUGIN_PORT>.
Use <0.0.0.0>:<PLUGIN_PORT> to
connect to the server running inside the plug-in from the Kubernetes host.
You can configure different ports for different applications in the given Kubernetes Jinja template under the ports section.
If you configure different port numbers in the Kubernetes Jinja template, you must hard code the port number and the corresponding port name in your respective applications.
Load the BYOI Custom Plug-in
Mount the BYOI custom plug-in image tar file and the modified Kubernetes YAML file to the Paragon Automation primary node and load the plug-in in the Paragon Automation management CLI.
Mount the BYOI plug-in image (compressed tar file) using the following command:
export HB_EXTRA_MOUNT1=/path/to/healthbot_plugin_name.tar.gzMount the Kubernetes YAML file using the following command:
export HB_EXTRA_MOUNT2=/path/to/healthbot_plugin_name.yml.j2Load the plug-in image and the Kubernetes YAML file using the following command
sudo -E healthbot load-plugin -i $HB_EXTRA_MOUNT1 -c $HB_EXTRA_MOUNT2You can see a confirmation message when the plug-in loads successfully.
-
(Optional) Select Configuration > Data Ingest > Settings > BYO Ingest Plugins page and view the custom plug-in in the Custom Plugins tab.
After you load your plug-in, create an instance of the custom BYOI plug-in in the Bring Your Own Ingest page. Since custom plug-ins do not use default Paragon Automation resources, you must configure a new rule and a playbook for the ingest plug-in.