TechLibrary

Table of Contents

Guide That Contains This Content

[+] Expand All

[-] Collapse All

Installing and Using Contrail Storage

Overview of the Contrail Storage Solution

Contrail provides a storage support solution using OpenStack Cinder configured to work with Ceph. Ceph is a unified, distributed storage system whose infrastructure provides storage services to Contrail. This Contrail storage solution provides a validated Network File System (NFS) storage service, however, it is not the Ceph FS distributed file system.

The Contrail storage solution has the following features:

Provides storage class features to Contrail clusters, including replication, reliability, and robustness.
Uses open source components.
Uses Ceph block and object storage functionality.
Integrates with OpenStack Cinder functionality.
Does not require virtual machines (VMs) to configure mirrors for replication.
Allows nodes to provide both compute and storage services.
Provides easy installation of basic storage functionality based on Contrail roles.
Provides services necessary to perform virtual machine migrations between compute nodes, and supports both migratable and non-migratable virtual machines.
Provides a Contrail-integrated user interface from which the user can monitor Ceph components and drill down for more information about components.

Basic Storage Functionality with Contrail

The following are basic interaction points between Contrail and the storage solution.

Cinder volumes must be manually configured prior to installing the Contrail storage solution. The Cinder volumes can be attached to virtual machines (VMs) to provide additional storage.
The storage solution stores virtual machine boot images and snapshots in Glance, using Ceph object storage functionality.
All storage nodes can be monitored through a graphical user interface (GUI).
It is possible to migrate virtual machines that have ephemeral storage in Ceph.

Ceph Block and Object Storage Functionality

Installing the Contrail storage solution creates the following Ceph configurations.

Each disk is configured as a standalone storage device, enhancing optimal performance and creating proper failure boundaries. Ceph allocates and assigns a process called object storage daemon (OSD) to each disk.
A replication factor of 2 is configured, consisting of one original instance plus one replica copy. Ceph ensures that each replica is on a different storage node.
A Ceph monitor process (mon) is configured on each storage node.
The correct number of placement groups are automatically configured, based on the number of disk drives in the cluster.
Properly identified SSD drives are set up for use as Ceph OSD journals to reduce write latencies.
An NFS server is created in a virtual machine within the cluster to support virtual machine migration. The NFS file system is mounted on all storage nodes, and every storage node has a shared Nova directory under the /var/lib/nova/instances directory. By default, this NFS file system is configured to utilize 30% of the total initial Contrail storage capacity.

Using the Contrail Storage User Interface

The Contrail storage solution provides a user interface integrated into the Contrail user interface. The storage solution user interface displays the following:

Customer usable space, which is different from Ceph total space. The displayed usable space does not display the space used by replication and other Ceph functions.
Monitor OSDs (disks), monitoring processes (MON), and state changes, enabling quick identification of resource failures within storage components.
Total cluster I/O statistics and individual drive statistics.
Ceph-specific information about each OSD (disk).
Ceph logs, Ceph nodes, and Ceph alerts.

Select Monitor > Infrastructure > Dashboard to display an at-a-glance view of the system infrastructure components, including the numbers of virtual routers, control nodes, analytics nodes, config nodes, and storage nodes currently operational, and a bubble chart of storage nodes showing the Available (%) and Total Storage (GB). See the following figure.

Bubble charts use the following color-coding scheme for storage nodes:

Blue—working as configured.
Red—error, node is down.
Yellow—one of the node disks is down.

Select Monitor > Storage > Dashboard to see a summary of cluster health, usage, pools, and disk status, and to gain insight into activity statistics for all nodes. See the following figure.

Hardware Specifications

The following are additional hardware specifications needed for the Contrail storage solution.

Additional minimum specifications:

Two 500 GB, 7200 RPM drives in the server 4 and server 5 cluster positions (those with the compute storage role) in the Contrail installation. This configuration provides 1 TB of clustered, replicated storage.

Recommended compute storage configuration:

For every 4-5 HDD devices on one compute storage node, use one SSD device to provide the OSD journals for that set of HDD devices.

Software Files for Compute Storage Nodes

The Contrail storage solution is only supported with the Ubuntu operating system.

For each compute storage node, ensure the following software is downloaded:

The storage Debian package: contrail-storage-packages_x.xx-xx~xxxxxx_all.deb.
NFS VM qcow2 image from Juniper Networks.

Contrail OpenStack Nova Modifications

Contrail's OpenStack Nova function has been modified to spawn both migratable and non-migratable virtual machines.

Nova's typical virtual machine storage directory, /var/lib/nova/instances, is used for non-migratable virtual machine ephemeral storage.
Contrail storage creates a new directory, /var/lib/nova/instances/global, used for the ephemeral storage for migratable virtual machines. The /var/lib/nova/instances/global must be mounted on a shared storage device (NFS with Contrail Storage), accessible from all the compute nodes.
To start a non-migratable virtual machine with the Nova CLI command nova boot, the additional argument "--meta storage_scope=local" must be provided.
To start a migratable virtual machine with nova boot, the additional argument "--meta storage_scope=global" must be provided. To force Nova and the Horizon UI to spawn migratable virtual machines by default, the storage scope must be set to global. This task is described in the next section.

Installing the Contrail Storage Solution

The Contrail storage solution can be installed using the same tools used to install Contrail, either by using Fabric (fab) commands or by using the Contrail Server Manager.

Both installation methods are described in the following sections.

Installation Notes

When installing a base operating system on any compute storage node, the operating system must be installed only on a single drive. The other drives must be configured as individual devices, and should not be concatenated together in a logical volume manager (LVM) device.
For best performance, it is recommended to use solid state devices (SSD) for the Ceph OSD journals. Each SSD device can provide OSD journal support to 3-6 HDD OSD devices, depending on the model of SSD device. Most SSD devices can support up to 4 HDDs, assuming the HDDs are running at capacity.

Using Fabric Commands to Install and Configure Storage

Use the information in this section to install storage using Fabric (fab) commands.

When installing the operating system on a compute storage node, install the operating system on a single drive and leave all other drives as unbundled.

Installing the Contrail storage solution with Fabric commands gives the following:

Base Ceph block device and object support.
Easy configuration of SSD devices for OSD journals.
Virtual machine migration support.
Limited Cinder multi-backend support.

Cautions

Before installing, ensure the following:

Manually ensure that the UID or GID of the Nova user is identical on all compute nodes before provisioning any software.
Manually ensure that the time is identical on all nodes by configuring NTP.

Fabric Installation Procedure

This section provides guidelines and steps for using Fabric (fab) commands to install the Contrail storage solution. The installation is similar to a regular Contrail fab installation, however, you define additional storage information in the testbed.py file, including:

Define new roles: storage-master and compute-storage.
Define how each additional non-root drive is used in the cluster.
Define potential additional virtual machine migration variables.
Copy and install the additional storage package to systems.

Install the storage Debian package on all nodes:
fab install_storage_pkg_all:/YYYY/contrail-storage-package-XXX.deb
After using the fab install_contrail command, use the fab install_storage command.
After using the fab setup_all command, use the fab setup_storage command.
If you need to enable Contrail-based live virtual machine migration, use the fab setup_nfs_livem command or the fab setup_nfs_livem_global command, as described in the following.
Note: If virtual machine migration is not needed, do not use either command.
- Use the fab setup_nfs_livem command to store the virtual machine's ephemeral storage on local drives.
- Use the fab setup_nfs_livem_global command to store the virtual machine's ephemeral storage within Contrail's storage (using Ceph). This command sets the cluster storage scope to global.
Add two new Contrail storage roles: compute-storage and storage-master.
- Define the storage-master role on all nodes running OpenStack. Although Ceph has no notion of a master, define this role because Ceph must be run on the node that runs the OpenStack software. OpenStack nodes typically do not have any cluster storage defined, only local storage.
- The storage-compute role is an add-on role. It means that compute nodes have the option of providing storage functionality. Standalone storage nodes are not supported.
Change the testbed.py file details as needed for your environment.
In the base configuration, define the storage_node_config values, which gives device details. See the following example.
```
storage_node_config = {

host2 : { 'disks' : ['/dev/sdb', '/dev/sdc'], 'ssd-disks': ['/dev/sdd', '/dev/sde'], 'local-disks' : ['/dev/sdf', '/dev/sdh'], 'nfs' : ['10.87.140.156:/test', '10.87.140.156:/test1']},

host3 : { 'disks' : ['/dev/sdb:/dev/sde', '/dev/sdc:/dev/sde', '/dev/sdd:/dev/sde', '/dev/sdf:/dev/sdj', '/dev/sdg:/dev/sdj', '/dev/sdh:/dev/sdj', '/dev/sdi:/dev/sdj', 'local-ssd-disks' : ['/dev/sdk', '/dev/sdl']},}
```
Available device details parameters include:
- disks and ssd-disks are Ceph disks.
- local-disk and local-ssd-disks are LVM disks.
- host2 in the example shows all the storage types that can be configured using Cinder multi-backend.
- disks is a list of HDD disks used for a Ceph HDD pool.
- ssd-disks is a list of SSD disks used for a Ceph SSD pool.
- local-disks is a list of disks used for local LVM storage.
- nfs is an NFS device.
- In the example, host3 is a more typical configuration.
- /dev/sde and /dev/sdj are SSD disks that are used as OSD journals for other HDD drives.
- local-ssd-disks is a list of disks used for local SSD LVM storage.

Add virtual machine migration as needed, using the following parameters.

live_migration = True

ceph_nfs_livevm = True

ceph_nfs_livem_subnet = ‘192.168.10.0/24’ # Private subnet to be provided for live migration VM

ceph_nfs_livem_image = ‘/Ubuntu/libmnfs.qcow2’ # path of live migration qcow2 image. This image is provided by Juniper Networks.

ceph_nfs_livem_host = host3 # host in which the NFS VM will run

For external NFS server-based live migration, use the following configuration.

live_migration = True

ext_nfs_livevm = True

ext_nfs_livem_mount = '10.10.10.10:/nfsmount' # External NFS server mount path

Note: When using an external NFS server, make sure the NFS server maps the uids and gids correctly, and provides read and write access for all the uids. If there is any issue related to the permission, either the VM launch errors out or the live migration fails with permission-related errors.

Using Server Manager to Install and Configure Storage

This section provides notes and guidelines to install the storage solution using the Contrail Server Manager. Installing the Contrail Storage solution using Server Manager provides:

Base Ceph block device and object support.
Easy configuration of SSD journals.
Support for live migration configuration.

Before installing the base operating system with Server Manager, ensure that the compute storage nodes have been configured with single operating system device installs.

Cautions

Virtual machine migration support uses a fixed IP address (192.168.101.3) for the livemnfs virtual machine.
There is no Cinder multi-backend support.
There is no support for single server provisioning, the entire cluster must be provisioned.

Server Manager Installation Procedure for Storage

This section provides notes and guidelines if you choose to install the storage solution using the Contrail Server Manager.

Upload the storage package: server-manager add image -f <filename.json>

where <filename.json> has content similar to the following example:

{

    "image": [

       {

            "id": "contrail-storage-packages_1.10-xx~xxxxxx_all",

            "parameters": "{}",

            "path": "/store/contrail-storage-packages_1.10-xx~xxxxxx_all.deb",

            "type": "contrail-storage-ubuntu-package",

            "version": "1.10-xx"

        }, 

    ]

}

Use the ceph-authtool command if you need to generate unique keys for administration, monitor, and OSD.
1. To install ceph-authtool on CentOS, use the following command:
  yum install http://ceph.com/rpm/el6/x86_64/ceph-common-0.80.5-0.el6.x86_64.rpm
2. To install ceph-authtool on Ubuntu:
  apt-get install ceph-common
3. Use the following command once for each key:
  ceph-authtool --gen-print-key
4. Add the generated keys to the cluster.json file:
```
cluster.json:

"storage_mon_secret":

"AQBDCdpTsB5FChAAOzI2++uosfmtj7tjmhPu0g==”,

"osd_bootstrap_key":

"AQBKCdpTmN+HGRAAl6rmStq5iYoPnANzSXLcXA==”,

"admin_key":

"AQBLCdpTuOS6FhAAfDW0SsdzyDAUeuwOr/h61A==”
```
5. In Contrail Release 2.10 and later, add live-migration configuration to the cluster.json file, if you are using live migration.
```
"live_migration": "enable",

"live_migration_nfs_vm_host":"compute-node-01",

"live_migration_storage_scope":"global",
```

Example: Configurations for Storage for Reimaging and Provisioning a Server

Use the following example configurations as guidelines for reimaging and provisioning a server for storage. Examples are given for configurations for releases prior to Release 2.10 and for configurations for Release 2.10 and later.

Define storage in the cluster. The following example configurations show new key-value pairs added to the configuration. The cluster section should appear similar to the following when storage is defined in a cluster.

Example: Storage and key-value pairs defined in releases prior to 2.10:

{

    "cluster" : [

        { 

            "id" : "demo-cluster",

            "parameters" : {

                "router_asn": "<asn>",

                "database_dir": "/home/cassandra",

                "database_token": "",

                "use_certificates": "False",

                "multi_tenancy": "False",

                "encapsulation_priority": "MPLSoUDP,MPLSoGRE,VXLAN",

                "service_token": "<password>",

                "keystone_user": "admin",

                "keystone_password": "<password>",

                "keystone_tenant": "admin",

                "analytics_data_ttl": "168",

                "subnet_mask": "<ip address>",

                "gateway": "<ip address>",

                "password": "<password>",

                "haproxy": "disable",

                "external_bgp": "",

                "domain": "demo.company.net",

"storage_mon_secret": "$ABC123”,

"osd_bootstrap_key": "$ABC123”,

"admin_key": "$ABC123”



            }

        }

    ]

}

Example: Storage and key-value pairs defined in releases 2.10 and later:

{

    "cluster" : [

        { 

            "id" : "demo-cluster",

            "parameters" : {

                "router_asn": "<asn>",

                "database_dir": "/home/cassandra",

                "database_token": "",

                "use_certificates": "False",

                "multi_tenancy": "False",

                "encapsulation_priority": "MPLSoUDP,MPLSoGRE,VXLAN",

                "service_token": "<password>",

                "keystone_user": "admin",

                "keystone_password": "<password>",

                "keystone_tenant": "admin",

                "analytics_data_ttl": "168",

                "subnet_mask": "<ip address>",

                "gateway": "<ip address>",

                "password": "<password>",

                "haproxy": "disable",

                "external_bgp": "",

                "domain": "demo.company.net",

"storage_mon_secret": "$ABC123”,

"osd_bootstrap_key": "$ABC123”,

"admin_key": "$ABC123”,

"live_migration" : "enable",

"live_migration_nfs_vm_host": "compute-host-01,

"live_migration_storage_scope": "global",

            }

        }

    ]

}

Add the disks key, the storage-compute role value, and the storage_repo_id key.

The storage_repo_id key must be added to servers with the storage-master or storage-compute roles.
The disks key-value pair must be added to servers with the storage-compute roles.
The storage-master value must be added to the roles key for the server that has the storage-master role.
The storage-compute value must be added to the roles key for the servers that have the storage-compute role.
The following server section is an example, showing the storage_repo_id and disks keys, and the storage-compute and storage-master values.
In the example, one server contains the storage-compute role and has 3 HDD drives (/dev/sdb, /dev/sdc, /dev/sdd), supporting 3 OSDs.
Each OSD uses one partition of an SSD drive (/dev/sde) as its OSD journal.
The server manager software correctly partitions /dev/sdd and assign one partition to each OSD. The storage_repo_id contains the base name of the Contrail storage package which has been added as an image to Server Manager.

Example: Server.json updates defined in releases earlier than 2.10:

{"server": [

        {

            "id": "demo2-server",

            "mac_address": "<mac address>",

            "ip_address": "<password>",

            "parameters" : {

                "interface_name": "eth1",

                "compute_non_mgmt_ip": "",

                "compute_non_mgmt_gway": "",

                "storage_repo_id":"contrail-storage-packages",

                "disks": ["/dev/sdb:/dev/sde", "/dev/sdc:/dev/sde", "/dev/sdd:/dev/sde"]

            },

            "roles" :

["config","openstack","control","compute","collector","webui","database","storage-compute","storage-master"],

            "cluster_id": "demo-cluster",

            "subnet_mask": "<ip address>",

            "gateway": "1<ip address>",

            "password": "<password>",

            "domain": "demo.company.net",

            "email": "id@company.net"

} ]

}

Example: Server.json updates defined in releases 2.10 and later:

 Server.json :

{"server": [

        {

            "id": "demo2-server",

            "mac_address": "<mac address>",

            "ip_address": "<ip address>",

            "parameters" : {

                "interface_name": "eth1",

                "compute_non_mgmt_ip": "",

                "compute_non_mgmt_gway": "",

                "storage_repo_id":"contrail-storage-packages",

                "disks": ["/dev/sdb:/dev/sde", "/dev/sdc:/dev/sde", "/dev/sdd:/dev/sde"]

            },

            "roles" :

["config","openstack","control","compute","collector","webui","database","storage-compute","storage-master"],

                "contrail": {

                    "control_data_interface": "p3p2"

               },
                "network": {
                "interfaces": [
                    {
                        "default_gateway": "<ip address>",
                        "dhcp": true,
                        "ip_address": "<ip address>",
                        "mac_address": "<mac address>",
                        "member_interfaces": "",
                        "name": "eth1",
                        "tor": "",
                        "tor_port": "",
                        "type": "physical"
                    },
                    {
                        "default_gateway": "<ip address>",
                        "dhcp": "",
                       "ip_address": "<ip address>",
                        "mac_address": "<mac address>",
                        "member_interfaces": "",
                        "name": "p3p2",
                        "tor": "",
                        "tor_port": "",
                        "type": "physical"
                    }
                ],
                "management_interface": "eth1"
            },

            "cluster_id": "demo-cluster",

            "subnet_mask": "<ip address>",

            "gateway": "<ip address>",

            "password": "<password>",

            "domain": "demo.company.net",

            "email": "id@company.net"

} ]

}

Use the following commands to provision the entire cluster:
# /opt/contrail/server_manager/client/server-manager -c
# /opt/contrail/server_manager/smgr_config.ini provision --cluster_id test-cluster contrail_test_pkg

Storage Installation Limits

General Limitations

Minimum number of storage nodes to configure: 2
The number of storage nodes should always be an even number (2, 4, 12, 22, etc.).

Fab Storage Install Limitations

There are no additional limitations to installation when using fab commands.

Server Manager Storage Install Limitations

There is no integrated way to add OSDs or drives to a storage node.
There is no integrated way to add new storage nodes to a cluster.
Provisioning a single server is not supported. You can add a server to Server Manager and then provision the entire cluster.
The live migration overlay network is preset to use 192.168.101.0/24.
The user must copy the image livemnfs.qcow2.gz to the folder /var/www/html/contrail/images before provisioning live migration using Server Manager.