Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

Contrail Cloud Software

 

This section provides detailed information about Contrail Cloud software components.

Contrail Cloud Software Overview

Juniper Networks Contrail Cloud is an integrated cloud platform that uses the Red Hat OpenStack Platform for orchestration, Juniper Contrail Networking for network automation, AppFormix for monitoring and analytics, and Red Hat Ceph Storage for storage. All Contrail Cloud software packages are deployed using a single installer. Contrail Cloud is commonly deployed in Service Provider and Enterprise environments.

Software Versions

This reference architecture supports Contrail Cloud 13.1 and 13.2.

The following software runs on the servers in this reference architecture.

Table 1: Contrail Cloud Release 13.2 Software Components

Software ComponentsContrail Cloud Release 13.2 Version

Contrail Networking

Contrail Networking Release 1910

AppFormix

AppFormix Release 3.1.6

OpenStack

Red Hat OpenStack 13 (z9)–OpenStack Queens Version (Red Hat CDN sync 1-October-2019)

Operating System and Kernel Versions

Red Hat Enterprise Linux 7.7 (RHEL 7.7), Kernel Version 3.10.0-1062.1.2.el7.x86_64 (Red Hat CDN sync 1-October-2019)

CEPH Storage

3.2 (Red Hat CDN sync 1-October-2019)

Table 2: Contrail Cloud Release 13.1 Software Components

Software ComponentsContrail Cloud Release 13.1 Version

Contrail Networking

Contrail Networking Release 1908

AppFormix

AppFormix Release 3.1

OpenStack

Red Hat OpenStack 13 - OpenStack Queens

Operating System and Kernel Versions

Red Hat Enterprise Linux 7.6 (RHEL 7.6), Kernel 3.10.0-957.5.1 (Red Hat OpenStack 13)

CEPH Storage

3.2

The software versions are automatically downloaded by the Contrail Cloud deployment scripts during installation. The software versions are fixed Contrail and Red Hat Openstack packages and containers downloaded from a Juniper Satellite.

You must send an email request to contrail_cloud_subscriptions@juniper.net to obtain the credentials to access the Juniper Satellite. See Deploying Contrail Cloud.

See Release Notes: Contrail Cloud for additional information regarding Contrail Cloud component versioning.

Node Types

Nodes in Contrail Cloud are software packages that run in servers and in virtual machines. Figure 1 illustrates the nodes that are deployed in Contrail Cloud 13.1.

Figure 1: Contrail Cloud Software Components
Contrail Cloud Software
Components

Table 3 provides a summary description of each node.

Table 3: Node Summary

NodeDescription

Compute node

Server provisioned to support virtualization; hosts virtual machines. Networking of the VMs on a compute node is provided by the Contrail Networking vRouter.

Control Host

Server that hosts controller nodes. Controller nodes are VMs responsible for controlling Contrail Cloud functions.

For additional information on control hosts and controller nodes, see Contrail Cloud Configuration.

Jumphost

A server that performs the following functions in Contrail Cloud:

  • hosts the undercloud. The undercloud runs as a VM on the jumphost and is responsible for provisioning and managing all controls hosts, compute nodes, and storage nodes.

  • hosts Contrail Cloud automation. All Contrail Cloud scripts and YAML configuration files are stored and instantiated on the jumphost.

  • hosts the Contrail Command web user interface virtual machine.

  • serves as the entry point for access to the undercloud or overcloud. You can “jump” between the overcloud and the undercloud on the jumphost.

Storage node

Server whose purpose is storing data. Storage nodes run Red Hat Ceph storage software in Contrail Cloud.

Note

Storage nodes must be separate from compute nodes in a Contrail Cloud environment. Hyperconverged nodes are not supported.

Note

TSN stands for ToR Service Node. A TSN is an optional node used when Contrail Networking is managing bare metal servers. In the current version of Contrail Networking, the TSN name has been changed to Contrail Service Node (CSN). The Red Hat OOO Heat templates, however, still contain the term TSN. Both terms are used in this guide.

Table 4 illustrates which nodes are deployed as virtual machines and which nodes must run in the operating system of a server.

Table 4: Virtual Machine Summary

FunctionDeployed as VM

Jumphost

No

 

Undercloud

Yes

Contrail Command

Yes

Controller Hosts

OpenStack

Yes

Contrail Controller

Yes

Contrail Analytics

Yes

Contrail Analytics DB

Yes

Appformix

Yes

Contrail Service Node (TOR Service Node)

Yes

Compute Nodes

vRouter in kernel mode

No

vRouter using DPDK

No

vRouter using SR-IOV

No

Storage Nodes (Ceph Software)

No

All hosts in a Contrail Cloud deployment must run on servers dedicated to the Contrail Cloud deployment since Contrail Cloud provisions the servers with a new operating system. The jumphost can only run the undercloud and Contrail Command VMs.

The hardware management for nodes deployed in bare-metal servers is typically done using IPMI while VMs are typically deployed using VirtualBMC. Note that a Red Hat support exception has been made to use VirtualBMC in Contrail Cloud production environments.

In the most common deployment architectures, an instance of each controller function runs in each of three controller node servers. This style of deployment is described in detail in this reference architecture. See Contrail Cloud Configuration.

Additional descriptions on how to configure variable deployments—such as splitting OpenStack, Contrail Controller, and Contrail Analytics onto separate sets of servers for improved performance and reliability—are also provided in this reference architecture. See Reference Architecture Variations.

Server Hardware

The servers used for the jumphost, controller nodes, compute nodes, and storage nodes have different physical server requirements.

Physical server requirements can vary between environments. The following table provides the recommended minimum physical server specifications for each node type.

Table 5: Physical Server Minimum Specifications

Role

Cores

Memory

Disk

Network

Jumphost

20 (2x10)

128 GB

2 x 1 TB HDD (SW RAID) — Host OS

Note: HW Raid recommended if RAID controller is available in your environment.

3 x 1 TB HDD — Data

1G — IPMI

1G/10G — Provisioning

1G/10G — Intranet

Control Host

36 (2x18)

256 GB

1 x 1 TB SSD (Cassandra Journaling)

Note: 1x1TB is the recommended disk memory option for control hosts.

2 x 1 TB HDD (HW RAID)—Host OS

3 x 2 TB HDD (HW RAID)—Data

1 x 1G — IPMI

1x1G/10G — Provisioning

1x1G/10G — Management (optional)

4 x 10G/25G/40G ports on two Intel Fortville (XL710/X710/XXV710) family NICs. Both high-speed NICs must be on NUMA 0.

Compute Node

28 (2x14)

256 GB

2 x 1 TB HDD (HW RAID) — Host OS

4 x 1 TB HDD — Data (only needed if Ceph storage not used to provide storage)

1 x 1G — IPMI

1x1G/10G — Provisioning

1x1G/10G — Management (optional)

4 x 10G/25G/40G ports on two Intel Fortville (XL710/X710/XXV710) family NICs. Both high-speed NICs must be on NUMA 0.

Storage Node

16 (2x8)

128 GB

2 x 1 TB HDD (HW RAID) — Host OS

2 x 480 GB SSDs — Ceph Journaling

10 x 1 TB HDDs — Ceph OSDs

1 x 1G — IPMI

1x1G/10G — Provisioning

1x1G/10G — Management (optional)

4 x 10G/25G/40G ports on two Intel Fortville (XL710/X710/XXV710) family NICs. Both high-speed NICs must be on NUMA 0.

Note

Virtual machines and processes located on NUMA 1 communicate with NICs via the QPI bus, which is less costly than balancing traffic across NICs in both NUMAs.

Note

For improved controller performance, you can use 6 SSDs. Configure 5 drives as RAID 5 and maintain a spare drive when using 6 SSDs.

Note

The number of drives and capacity in storage nodes can be increased as needed. The Red Hat recommendation of a 1:5 ratio of SSD:HDD drives should be followed. For additional information, see the Throughput-optimized Solutions section in the Red Hat Storage Hardware Selection Guide).

The sizing and configuration recommendations provided in this reference architecture are intended to enable a Contrail Cloud environment to operate at the highest possible performance. The configuration file examples provided in this reference architecture assume these server configuration recommendations are followed. Other server configurations are supported; see Reference Architecture Variations.

Networking for Contrail Cloud 13.1 and 13.2

Traffic for different purposes is segregated onto different physical logical interfaces in Contrail Cloud 13.1 and 13.2. The configuration of server-facing ports in the IP Fabric must match the configuration of the corresponding server ports as specified in the Contrail Cloud configuration files.

Table 6 lists the server-layer networks used in Contrail Cloud 13.1 and 13.2.

Table 6: Networking in Contrail Cloud Summary

NetworkPurpose

IPMI

Provides hardware management of servers.

IPMI services are mostly used by the Openstack Ironic service and are established outside of the Contrail Cloud installation. IPMI services can be used by the server nodes in Contrail Cloud.

Intranet

Provides user access to the jumphost.

Provides access to satellite repositories for the jumphost.

Provides access to the Contrail Command web user interface.

Provides external network access via SNAT from the control plane network.

Provisioning or Control Plane

Deploys new nodes using PXE booting and to install software packages on overcloud bare metal servers. The provisioning/control plane network is used by Red Hat Director and is predefined before the creation of the undercloud.

Internal API

Provides communication with the OpenStack and Contrail Networking services, including Keystone, Nova, and Neutron using API communication, RPC messages, and database communication.

External

Provides tenant administrators access to the OpenStack Dashboard (Horizon) graphic interface, the public APIs for Openstack services, public Contrail APIs, and the Appformix WebUI. Commonly used as a path to the intranet.

Tenant

Supports overlay data-plane traffic—VXLAN and MPLS over UDP—and Contrail Controller to Contrail vRouter control plane traffic.

Storage

Supports storage data traffic for Ceph, block storage, NFS, iSCSI, and any other storage types.

Storage Management

Provides Ceph control, management, and replication traffic.

Management

(Optional) Provides direct access to compute nodes without having to send traffic through the jumphost.

Can be used for DNS and NTP services.

Note

Depending on the context, Red Hat documentation uses the names Provisioning and Control Plane for the same network. In this document, the Provisioning network name is used to refer to the network in the undercloud and the Control Plane network name is used to refer to the network in the overcloud. The Control Plane network name, notably, is used in configuration files.

All networks, except IPMI and Intranet, are configured on nodes by Contrail Cloud 13.1 and 13.2 using information contained in YAML configuration files. Table 7 lists the networking requirements by role in Contrail Cloud 13.1 and 13.2.

Table 7: Device & VM Network Summary

RoleIntranetProvisioningInternal APIExternalTenantStorageStorage MgmtMgmt

Jumphost (Undercloud VM)

Openstack Controller

 

Optional

Optional

Contrail Controller

Optional

Optional

Contrail Analytics

Optional

Optional

Contrail Analytics DB

Optional

Optional

Contrail Service Node (ToR Service Node)

Optional

Optional

Appformix Controller

Optional

Optional

Control Host (Running all Controller VMs)

Optional

Compute Node

Optional

Storage Node

Optional

Table 8 provides the networking requirements for each host type in a Contrail Cloud. The table assumes that each listed host is running all applicable controller functions.

Table 8: Host Network Summary

HostIntranetProvisioningInternal APIExternalTenantStorageStorage MgmtMgmt

Jumphost

Control Host

 

Optional

Compute Node

Optional

Storage Node

   

Optional

The server port assignments depend on the available NICs in each server.

The use of a management network to enable direct access to nodes without going through the jumphost is optional. It is included in this reference architecture for convenience.

The diagram, below, shows the layout for a Contrail Cloud environment in three racks, with controller nodes that are running all controller functions.

Figure 2: High-level Architecture Overview—Rack View
High-level Architecture Overview—Rack View
Note

Red Hat configuration files use the term leaf to refer to a group of servers—generally in the same rack—which share switches for network connectivity. In this reference architecture, connectivity for servers in the same Red Hat leaf is implemented by a pair of switches with leaf roles in the IP Fabric and by a management switch.

Note

Some Contrail Cloud documentation shows the controller nodes all placed in the same rack. The nodes are placed in separate racks for resilience in this reference architecture. The architecture shown in the other Contrail documentation is a supported variance for this reference architecture. See Reference Architecture Variations.

The IPMI, Management, and Provisioning networks are connected via management switches and the networks are stretched between switches using trunk connections for VLANs, or by using a separate VXLAN for each network.

The external network contains the externally-routable API and UI IP addresses for the various controller functions, and generally these addresses reside in the same subnet. The VLAN for the external network is configured to be in a VXLAN, which terminates in a VRF which is configured to route between the network where tenant user packets arrive from the external network. The network naming convention for these rack-specific subnets is network_name(leaf #), where the leaf number is appended to the network name. The name of the network without an appended leaf number is used to identify the subnets that are used by control hosts.

The rest of the networks are connected into the leaf switches within the IP Fabric, where EVPN-VXLAN is used to connect switches between racks. The diagram shows connectivity for these networks within supernets that span racks, but these supernets are subdivided on a per rack basis using the leaf-id as part of each subnet identifier. This setup is described in more detail later in this document.

The other networks connected to the IP Fabric have a separate subnet for each rack. Routing is used to connect the subnets together using VXLAN. The routing can be configured in leaf devices (edge routing) or spine devices (central routing). Edge routing is used in this reference architecture.

Table 9 summarizes recommended network configurations options.

Table 9: Recommended Network Subnets and Implementations

NetworkSubnet DescriptionImplementation

IPMI

IPMI is typically an external service that can be used by devices that can route to it. IPMI is not established by Contrail Cloud.

IPMI is typically not spanned over racks. It can be spanned over racks using VXLAN.

Intranet

Administrative access to jumphost.

Specific port configuration

Provisioning

Layer 2 single subnet

Span over racks using VLAN or VXLAN

Internal API

Layer 2 single subnet

Span over racks using VLAN or VXLAN

Internal API leaf-number

One subnet per rack

leaf-number is the Red Hat Openstack leaf number.

Layer 3 routing between racks

External

Layer 2 single subnet

Span over racks using VLAN or VXLAN

Tenant

Layer 2 single subnet

Span over racks using VLAN or VXLAN

Tenant leaf-number

One subnet per rack

leaf-number is the Red Hat Openstack leaf number.

Layer 3 routing between racks

Storage

Layer 2 single subnet

Span over racks using VLAN or VXLAN

Storage leaf-number

One subnet per rack

leaf-number is the Red Hat Openstack leaf number.

Layer 3 routing between racks

Storage Management

Layer 2 single subnet

Span over racks using VLAN or VXLAN

Storage Management leaf-number

One subnet per rack

leaf-number is the Red Hat Openstack leaf number.

Layer 3 routing between racks

Management

Layer 2 single subnet

Span over racks using VLAN or VXLAN

When a network is composed of multiple subnets—one subnet per rack—the subnet that contains all the rack subnets is termed the supernet. The subnet address of the supernet is used to configure static routes on servers to ensure proper traffic flow. It is important that each supernet address range is large enough to accommodate future growth in the number of racks.

For networks that have a subnet per rack, the base subnet that is used for control hosts is stretched across the racks containing such nodes. A naming convention is used in configuration files where the name of the subnet in each rack has a suffix of the number of the rack (leaf in Red Hat terminology) and the base subnet has no suffix. For instance, the tenant network is stretched across the racks containing control hosts, and tenant0, tenant1, and is used for compute and storage nodes in each rack. This is explained in detail in Internal API, Tenant, Storage, and Storage Management Network sections.

The network configuration described in the table is designed to provide maximum robustness and performance. Variations are supported where networks span racks instead.

The upcoming diagrams illustrate how VLANs, VTEPs, IRBs, and VRFs are configured in gateway, spine, and leaf devices. Illustrations of these components in edge-routed bridging (ERB) environments—where routing is done on the leaf devices—and centrally-routed bridging (CRB)—where routing is done on the spine devices—are provided.

For additional information on ERB and CRB architectures, see Centrally-Routed Bridging Overlay Design and Implementation and Edge-Routed Bridging Overlay Design and Implementation in the Cloud Data Center Architecture Guide.

Figure 3: Network Connectivity Details
Network Connectivity Details
Note

The diagram shows a logical view of the networks. All traffic between leaf switches passes through a spine switch and not directly between leaf switches.

Figure 4: VXLAN Connectivity for a Leaf Switch in an ERB Environment
VXLAN Connectivity for a Leaf Switch in an ERB Environment

The Virtual Routing and Forwarding (VRF) instances are created on the spine devices and provide VXLAN tunnels to each leaf device when centralized routing and bridging (CRB) is used.

Figure 5 illustrates VXLAN connectivity for a leaf device in a CRB environment.

Figure 5: VXLAN Connectivity for a Leaf Switch in a CRB Environment
VXLAN Connectivity for a Leaf
Switch in a CRB Environment

The main features of this network design:

Table 10: VLAN Summaries

DeviceVLAN Summary

Management Switch

The IPMI, Management, Provisioning, and Intranet networks all connect the management switch to a server on different ports.

The port traffic for each network arrives on the management switch untagged and is configured into the same VLAN.

The VLAN is extended between the management switches on different racks.

Leaf Switch (EVPN-VXLAN Fabric)

Traffic from the external, internal API, tenant, storage, and storage management networks arrives on the leaf switch’s high-speed interfaces from the servers in different VLANs.

The VLANs are configured in VTEPs.

Leaf ports are configured with logical interfaces that have VLANs for the networks that will be attached by servers to that port.

Each VLAN on each switch is configured into a VXLAN VTEP and EVPN advertises host routes for each connected server to the spines.

The VLANs used for each network are specified in the file overcloud-nics.yml.

Spine Switch (EVPN-VXLAN Fabric)

The spine switches are configured with a VTEP for each of the Internal API, Tenant, Storage and Storage-Mgt networks, and each of these are connected to an IRB interface whose IP address is that of the supernet. Each spine switch has a VRF that receives routes to each host from the leaf switches.

SDN Gateway Router

The SDN gateways are configured with a VTEP and VRF for the external network. Each SDN gateway will advertise a route for the external network to peers outside the Layer 2 network.

This reference architecture does not provide fabric configuration options and details.

To configure an EVPN-VXLAN fabric that can be used by the nodes in a Contrail Cloud to transport Layer 2 and Layer 3 traffic, see Data Center Fabric Architecture Guide.

Interface Addressing and Settings

This section covers interface addressing and other interface settings.

Leaf Subnets and Supernets

Leaf subnet addresses are contained within a supernet address range. The supernet range should be large enough to contain the largest possible number of leaf subnet addresses—which is the number of racks— that could be deployed in your Contrail Cloud environment. Set your supernet address ranges thoughtfully, as you cannot change the values of supernet address ranges after the first servers are deployed.

The spans of the various networks are shown below (assuming that /24 networks are used for leaf subnets, and a /16 network is used for each supernet.

Figure 6: Leaf Subnets Within Supernets
Leaf Subnets Within Supernets

​As an example, if the supernet for the internal_api network is 172.18.0.0/16 then the leaf subnets could be 172.18.0.0/24 (Leaf0), 172.18.1.0/24 (Leaf1), and 172.18.2.0/24 (Leaf2).

The subnet addresses used by control hosts are put at the high end of the supernet range in order to allow synchronization of the leaf devices with their subnet addresses.

Address Allocation Within Networks

Contrail Cloud networks are subdivided into groups of addresses for different purposes. Only part of the subnet is used. See Figure 7.

Figure 7: Subnet Address Spaces by Network
Subnet Address Spaces
by Network

The DHCP range for the provision and management networks is large enough—it uses a /23 CIDR—to supply addresses to all of the controller VMs and all of the compute and storage nodes. in the Contrail Cloud environment. The Internal API network needs to supply addresses only to control hosts and the VMs running on them, so its DHCP range is small. The external subnet is typically allocated by a datacenter administrator and only has to accommodate addressing for the controller functions that have external access. The VIPs that will be the public access IP addresses for GUIs and APIs, and AppFormix VM addresses are statically configured outside the DHCP range of the external network. The base subnets for the internal_api and storage_mgmt networks have a small DHCP range and a VIP is allocated outside that range. The leaf-specific subnets for internal_api and storage_mgmt networks and all the subnets for tenant and storage networks simply have a DHCP range allocated that is large enough to accommodate all the servers that can be placed in one rack.

The address of the default gateway is usually the first or last address of a subnet. The exception is the external network, whose default gateway is determined by the data center architecture.

Use of IPv6 in Contrail Cloud networks

Red Hat supports IPv6 in all networks. Contrail Cloud does not support IPv6 in the provision, internalAPI, and tenant networks, however, because the Contrail Cloud vRouter does not support IPv6 for compute node addresses and some Contrail API components.

MTU settings

Packets passing between VMs are encapsulated in MPLSoUDP, MPLSoGRE, or VXLAN headers. These headers increase packet sizes. MTU settings in configuration files must account for these packet size increases.

Table 11 provides the MTU setting requirements when the switching environment supports jumbo frames of 9000 bytes.

Table 11: MTU Setting Requirements

EntityMTU setting

Switch ports

9216

Interfaces on bare metals servers Controllers VMs defined by overcloud networks

9100

Tenant VM interface

1500 or 9000

The difference in MTU between the various entities is more than actually needed to avoid fragmentation. These MTU settings can be tuned more precisely, if needed for your environment.

If jumbo frames cannot be configured in the switch ports, then the MTU on server ports and tenant VM interfaces should be reduced to allow passage of VXLAN and Ethernet headers without fragmentation (typically, 1400 bytes for server ports and 1300 bytes for tenant VM interfaces).

Note

Due to limitations in the Linux ip netns feature, if SNAT or LBaaS are used with DPDK, the MTU must be set to less than 1500 bytes. This limitation is documented in the Contrail 1910 release notes as issue JCB-177787.

Interface Naming Conventions

The examples in this reference architecture use a numeric convention instead of named interfaces to identify interfaces. This convention allows the same configuration file snippets to manage servers of different types in scenarios where the interface names seen in BIOS and OS differ.

Table 12 provides a naming schema to use if interface names are used in place of numeric conventions.

For example, a server with 6 interfaces— 2 1G interfaces and 4 10G interfaces—could use the following names and corresponding “numeric” names.

Table 12: Interface Naming Convention Schema Example—2 1G and 4 10G interfaces.

Interface NameNumeric Name

eno1

nic1

eno2

nic2

ens7f0

nic3

ens7f1

nic4

ens7f2

nic5

ens7f3

nic6

Note

The number of NICs correspond to the physical order of NICs in a server (PCI Address).

Note

Red Hat numeric naming uses the term nicN for each interface. Physical NICs usually contain two or four interfaces per NIC.

After the introspection phase (running ./inventory-assign.sh), the names and order of interfaces on a server can be found by logging into the undercloud VM and running the openstack baremetal introspection interface list node-name command: