Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

 

Aggregate Discovery and Alarms with OpenStack Heat Services

 

Contrail Insights integration with Heat has two independent aspects: discovery and alarms.

Heat Stack Discovery

The Contrail Insights OpenStack Adapter uses Heat APIs to discover Heat stacks in an OpenStack cluster. Each stack is represented as an aggregate in Contrail Insights with the label Heat. When OpenStack Adapter discovers a Heat stack, OpenStack Adapter configures an aggregate in the Contrail Insights Platform, and adds any virtual machines and virtual networks defined by the Heat stack as members of the aggregate. See Figure 1.

Figure 1: Heat Stack Discovery in Contrail Insights
Heat Stack Discovery in Contrail
Insights

Discovery functionality is provided by Contrail Insights OpenStack Adapter as part of the standard OpenStack integration. Discovery does not require any modifications to the OpenStack controller, OpenStack configuration, or Heat service, and does not require installation of the Contrail Insights Heat plug-in.

Figure 2 shows two Heat stacks that were discovered by Contrail Insights: stack1 and ubuntu_stack. Each stack is represented in Contrail Insights as a Mixed aggregate. A Mixed aggregate may contain entities of different types, such as virtual machine and virtual network. To view aggregate Heat stacks, select Dashboard, then from the Infrastructure tab, select Aggregates.

Figure 2: Mixed Aggregate Heat Stacks in Contrail Insights
Mixed Aggregate Heat
Stacks in Contrail Insights

Figure 3 shows the entities in the Heat stack ubuntu_stack. There are two virtual machines, ubuntu_vm1 and ubuntu_vm2, and one virtual network, ubuntu_network1. To view this page, select Dashboard, then from the Infrastructure tab, select Aggregates. From the Aggregate Select One tab, select the entity to view.

Figure 3: Virtual Machines in Heat Stacks in Contrail Insights
Virtual Machines
in Heat Stacks in Contrail Insights

Contrail Insights Alarm Resource Types

In addition to discovery of Heat stacks, Contrail Insights provides a Heat plug-in that defines two new resource types for Heat templates.

OS::AppFormix::AlarmThis resource type is used to define single alarms for monitoring resources in a Heat stack.
OS::AppFormix::CompositeAlarmThis resource type is used to define composite alarms for monitoring resources in a Heat stack.

The alarms and composite alarms are configured in Contrail Insights when a Heat stack is created from the template, and are evaluated by the Contrail Insights stream-based, distributed analysis engine.

You benefit by maintaining monitoring configuration in the same template that defines the resources (for example, virtual machines) to be monitored. Contrail Insights alarms and composite alarms are configured at the time that resources in the Heat stack are instantiated. Further, these alarms can be used to trigger scale-up and scale-down of Heat AutoScaling policies, which enables Heat to react more quickly and accurately due to the responsive and fine-grained Contrail Insights alarms.

Figure 4: Interaction Sequence between Heat, Contrail Insights Heat Plug-In, and Contrail Insights Platform
Interaction Sequence between Heat, Contrail Insights Heat Plug-In,
and Contrail Insights Platform

When a Heat stack is created using a Heat template that has an OS::AppFormix::Alarm or OS::AppFormix::CompositeAlarm resource, Heat will pass the resource properties to Contrail Insights Heat plug-in to configure the alarm in Contrail Insights. Figure 4 illustrates the interaction between Heat, Contrail Insights Heat plug-in, and Contrail Insights Platform in the following sequence of events.

  1. User instantiates a Heat stack from a template with an OS::AppFormix::Alarm or OS::AppFormix::CompositeAlarm resource.

  2. Heat passes the alarm properties to Contrail Insights Heat plug-in.

  3. When the resource OS::AppFormix::Alarm is used, Contrail Insights Heat plug-in configures an alarm in Contrail Insights Platform using the Contrail Insights REST API. The URL for Contrail Insights Platform is a configuration parameter (see “Install Contrail Insights Heat Plug-in”). The alarm is configured with mode set to Event. The alarm will generate a notification for each interval in which the condition for the alarm is satisfied.

  4. When the resource OS::AppFormix::CompositeAlarm is used, Contrail Insights Heat plug-in configures a composite alarm in Contrail Insights Platform using the Contrail Insights REST API. The user can define multiple individual alarms in the composite alarm. The state of the composite alarm is a combination of the states of the individual alarms. The user can define weights for the individual alarms and a threshold for the composite alarm. The composite alarm is active when the sum of the weights of the active alarms equals or exceeds the user-defined threshold (see “Example: Heat Autoscaling with OS::AppFormix::Alarm”). A notification will be generated every 60 seconds for as long as the composite alarm is active.

  5. When the alarm or composite alarm triggers, a notification is delivered by HTTP/HTTPS POST to an endpoint specified in the notification_url property of the alarm. To enable auto-scaling, a Heat template can specify the signal_url of a Heat ScalingPolicy resource as the notification_url. In that case, the notification is sent to Heat for processing.

To make the OS::AppFormix::Alarm and OS::AppFormix::CompositeAlarm resource types available to Heat templates, the Contrail Insights Heat plug-in must be installed and configured on the OpenStack controller host(s). See the following section, “Install Contrail Insights Heat Plug-in.”

For more information about the extensible design of Heat resources using plug-ins, refer to Heat documentation.

Install Contrail Insights Heat Plug-In

To install and configure the Heat plug-in on the OpenStack controller host:

  1. Copy appformix-openstack package.

    Copy the appformix_openstack Python package from the release bundle to the OpenStack controller host on which Heat service is running. This package is provided as either a Python wheel or an RPM package in the pkg directory of the release bundle.

  2. Install appformix-openstack package.

    On the OpenStack controller host that runs the Heat service, install the appformix-openstack package. The latest version of appformix-openstack package is 0.6.2.

    By default, this will install the resources in:

    If the OpenStack services are running in containers, the resources should be installed in a directory that is accessible to the Heat containers. Use the --target option with pip install to install the resources in a different directory. For example:

  3. Modify the Heat configuration file.
    • Define a variable called appformix_controller_url in the [DEFAULT] section and set it to the base URL of the Contrail Insights Platform.

    • Add the installation directory to the list of plug-in directories. Look for the plugin_dirs entry in the [DEFAULT] section and add the installation directory to the end of the list. If the OpenStack services are running in containers, specify the mount path of the installation directory inside the Heat containers.

    • If desired, define variables called appformix_task_num_iterations and appformix_task_wait_milliseconds to control how many times and how frequently the Heat plug-in checks the status of an Contrail Insights API request before declaring that the operation has timed out. Both variables accept Integer values. If these variables are undefined, they default to the following values:

    This is what the heat.conf file should look like after modification:

    If the OpenStack services are running in containers, make sure the changes are made in the heat.conf files in all the Heat containers.

  4. Restart all the OpenStack Heat services.

    If the OpenStack services are running in containers, restart all the Heat containers.

OS::AppFormix::Alarm Configuration in Heat Template

The OS::AppFormix::Alarm resource type can be used in Heat templates to create a Contrail Insights alarm. The resource type has the following input parameters:

Table 1: OS::AppFormix::Alarm Resource Type Input Parameters

Parameter

Description

alarm_name

A name that identifies the alarm.

alarm_metric

Metric to evaluate in the alarm.

To see a list of choices, use the Contrail Insights API endpoint /describe/alarms and look for the following list in the output: output['EventRuleParams']['MetricTypeMap'][0]['static']['instance']. Use the value in the Value key as the metric name.

output['EventRuleParams']['MetricTypeMap'][0]['static']['instance']

Use the value in the Value key as the metric name.

threshold

Value by which to compare a metric measurement. Units for the threshold depend on the value of alarm_metric.

aggregation_function

Operation to use for combining measured values before comparison. Choices are:

  • average

  • max

  • min

  • std-dev

  • sum

comparison_function

Operation to use for comparing measured values to the threshold. Choices are:

  • below

  • equal

  • above

  • increasing-at-a-minimum-rate-of

  • decreasing-at-a-minimum-rate-of

duration

Number of seconds for which sample values will be collected before being combined.

num_intervals

Number of intervals of length duration for which data will be collected before comparison.

num_exception_intervals

Number of intervals of length duration for which the alarm condition has to be true for the alarm to be considered active.

project_id

(Optional) ID of a project that contains the instances on which the alarm should be evaluated.

aggregate_id

(Optional) ID of an aggregate that contains the instances on which the alarm should be evaluated.

Use the following syntax to indicate that alarm should be evaluated on instances in the current Heat stack:

aggregate_id: { get_param: "OS::stack_id" }

Either project_id or aggregate_id must be specified in the template.

notification_url

URL to which a notification will be sent when the alarm is active. This is any URL prepared to receive notification from Contrail Insights (refer to Notifications). For Heat templates that use Contrail Insights Alarms to trigger autoscaling, this URL should be set to the signal_url of the scaling policy (see example in “Example: Heat Autoscaling with OS::AppFormix::Alarm”).

Example: Heat Autoscaling with OS::AppFormix::Alarm

With Contrail Insights Heat plug-in, Contrail Insights Alarms can be used in Heat Autoscaling templates in place of Ceilometer Alarms. The following Heat template uses OS::AppFormix::Alarm to automatically scale the number of running instances based on CPU utilization:

The following sequence describes what happens when a Heat stack is created from the “Heat Autoscaling with OS::AppFormix::Alarm” template.

  • When a heat stack is deployed using this template, a single instance test_vm is initially created.

  • The two Contrail Insights alarms cpu_alarm_high and cpu_alarm_low are used to monitor CPU utilization on the instance. They can be defined to monitor any metric that Contrail Insights collects.

  • When the CPU utilization on the instance goes above 80 percent, the alarm cpu_alarm_high is triggered. This results in the execution of the scaleup_policy which increases the number of running instances by 1 every 60 seconds, for as long as the alarm is active. The scaleup_policy stops executing when the number of running instances equals the value in max_size.

  • When the CPU utilization on the instance drops below 10 percent, the alarm cpu_alarm_low is triggered. This results in the execution of the scaledown_policy which decreases the number of running instances by 1 every 300 seconds, for as long as the alarm is active. The scaledown_policy stops executing when the number of running instances equals the value in min_size.

Create a Heat Stack for the Auto-Scaling Template using OS::AppFormix::Alarm

Now, let's create a Heat stack from the template “Example: Heat Autoscaling with OS::AppFormix::Alarm,” and observe what happens when we add CPU load on the VM to trigger the Contrail Insights Alarm.

  1. Save the template defined in “Example: Heat Autoscaling with OS::AppFormix::Alarm” in a file named appformix_autoscaling.yaml. Enter appropriate values in the project_idor aggregate_id fields. Then create a Heat stack using the template:
  2. Check that there is a single test_vm instance running.
  3. Generate some load on test_vm. Watch for the cpu_alarm_high alarm to become active on the Contrail Insights Dashboard.
  4. When the alarm is active, check the running instances on the cluster. There should now be two running instances called test_vm.
  5. Stop the load generator on test_vm. Watch for the cpu_alarm_low alarm to become active on the Contrail Insights Dashboard.
  6. When the alarm is active, check the running instances on the cluster. There should now be a single test_vm instance running.

OS::AppFormix::CompositeAlarm Configuration in Heat Template

You can define multiple individual Alarms and combine them in a Composite Alarm. The state of the Composite Alarm is a combination of the states of the individual Alarms. You can define weights for the individual Alarms and a threshold for the Composite Alarm. The Composite Alarm is active when the sum of the weights of the active Alarms equals or exceeds the user-defined threshold.

The OS::AppFormix::CompositeAlarm resource type can be used in Heat templates to create a Contrail Insights Composite Alarm. The resource type has the following input parameters:

Table 2: OS::AppFormix::CompositeAlarm Resource Type Input Parameters

Parameter

Description

composite_alarm_name

A name that identifies the composite alarm.

project_id

(Optional) ID of a project that contains the instances on which the composite alarm should be evaluated.

aggregate_id

(Optional) ID of an aggregate that contains the instances on which the composite alarm should be evaluated.

Use the following syntax to indicate that the alarm should be evaluated on instances in the current Heat stack:

aggregate_id: { get_param: "OS::stack_id" }

Either project_id or aggregate_id must be specified in the template.

notification_url

URL to which a notification will be sent when the alarm is active. This is any URL prepared to receive notification from Contrail Insights (refer to Notifications). For Heat templates that use Contrail Insights Alarms to trigger autoscaling, this URL should be set to the signal_url of the scaling policy (see “Example: Heat Autoscaling with OS::AppFormix::CompositeAlarm”).

composite_alarm_rules

List of individual alarms that make up the composite alarm. Each individual alarm has the same input parameters as the OS::AppFormix::Alarm resource, except for notification_url, project_id and aggregate_id. These parameters are defined once for the entire composite alarm. Each individual alarm also has an input parameter called alarm_weight, which is a value between 0 and 1. The weights of all active alarms are summed and compared to the value of composite_alarm_thresholdto determine if the composite alarm is active.

composite_alarm_threshold

Value between 0 and 1 used to determine if composite alarm is active. If the sum of weights of all active rules equals or exceeds the threshold, the composite alarm is determined to be active.

Example: Heat Autoscaling with OS::AppFormix::CompositeAlarm

With Contrail Insights Heat plug-in, Contrail Insights Composite Alarms can be used in Heat Autoscaling templates in place of Ceilometer Alarms. The following Heat template uses OS::AppFormix::CompositeAlarm to automatically scale the number of running instances based on CPU utilization and memory utilization.

The following sequence describes what happens when a Heat stack is created from the above template.

  • When a heat stack is deployed using this template, a single instance test_vm is initially created.

  • The two Contrail Insights composite alarms composite_alarm_high and composite_alarm_low are used to monitor resource utilization on the instances in the Heat stack. They are comprised of two individual alarms to monitor CPU utilization and memory utilization on the instances. They can be defined to monitor any metric that Contrail Insights collects.

  • The individual alarms in composite_alarm_high are defined with weights of 0.5 each and the composite alarm is defined with a threshold of 0.5. This means that the composite alarm will be considered active when any of the individual alarms is active.

  • The individual alarms in composite_alarm_low are defined with weights of 0.5 each and the composite alarm is defined with a threshold of 1.0. This means that the composite alarm will be considered active only when both the individual alarms are active.

  • When either the CPU utilization or memory utilization on the instance test_vm goes above 80 percent, the composite alarm composite_alarm_high is triggered. This results in the execution of the scaleup_policy which increases the number of running instances by 1 every 60 seconds for as long as the composite alarm is active. The scaleup_policy stops executing when the number of running instances equals the value in max_size.

  • When both the CPU utilization and memory utilization on the instance test_vm drops below 10 percent, the composite alarm composite_alarm_low is triggered. This results in the execution of the scaledown_policy which decreases the number of running instances by 1 every 60 seconds, for as long as the composite alarm is active. The scaledown_policy stops executing when the number of running instances equals the value in min_size.

Create a Heat Stack for the Auto-Scaling Template using OS::AppFormix::CompositeAlarm

Now, let's create a Heat stack from the template “Example: Heat Autoscaling with OS::AppFormix::CompositeAlarm” and observe what happens when we add CPU load on the VM to trigger the Contrail Insights Composite Alarm.

  1. Save the template defined in “Example: Heat Autoscaling with OS::AppFormix::CompositeAlarm” in a file called appformix_composite_autoscaling.yaml. Enter appropriate values in the project_id or aggregate_id fields. Then create a Heat stack using the template:
  2. Check that there is a single test_vm instance running.
  3. Generate some load on test_vm. Watch for the composite_alarm_high composite alarm to become active on the Contrail Insights Dashboard.
  4. When the composite alarm is active, check the running instances on the cluster. There should now be two running instances called test_vm.
  5. Stop the load generator on test_vm. Watch for the composite_alarm_low composite alarm to become active on the Contrail Insights Dashboard.
  6. When the alarm is active, check the running instances on the cluster. There should now be a single test_vm instance running.

Troubleshooting

For debugging, enable the verbose and debug options by adding them to the [DEFAULT] section in heat.conf.

Then restart the Heat services or containers. Detailed logs will appear in /var/log/heat/heat-engine.log.

Related Documentation