Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

Navigation
Guide That Contains This Content
[+] Expand All
[-] Collapse All

    Installing and Using Contrail Storage

    Overview of the Contrail Storage Solution

    Starting with Contrail Release 2.00, Contrail provides a storage support solution using OpenStack Cinder configured to work with Ceph. Ceph is a unified, distributed storage system whose infrastructure provides storage services to Contrail. The Contrail solution provides a validated Network File System (NFS) storage service, however, it is not the Ceph FS distributed file system.

    The Contrail storage solution has the following features:

    • Provides storage class features to Contrail clusters, including replication, reliability, and robustness.
    • Uses open source components.
    • Uses Ceph block and object storage functionality.
    • Integrates with OpenStack Cinder functionality.
    • Does not require virtual machines (VMs) to configure mirrors for replication.
    • Allows nodes to provide both compute and storage services.
    • Provides easy installation of basic storage functionality based on Contrail roles.
    • Provides services necessary to perform virtual machine migrations between compute nodes, and supports both migratable and non-migratable virtual machines.
    • Provides a Contrail-integrated user interface from which the user can monitor Ceph components and drill down for more information about components.

    Basic Storage Functionality with Contrail

    The following are basic interaction points between Contrail and the storage solution.

    • Cinder volumes must be manually configured prior to installing the Contrail storage solution. The Cinder volumes can be attached to virtual machines (VMs) to provide additional storage.
    • The storage solution stores virtual machine boot images and snapshots in Glance, using Ceph object storage functionality.
    • All storage nodes can be monitored through a graphical user interface (GUI).
    • It is possible to migrate virtual machines that have ephemeral storage in Ceph.

    Ceph Block and Object Storage Functionality

    Installing the Contrail storage solution creates the following Ceph configurations.

    • Each each disk is configured as a standalone storage device, enhancing optimal performance and creating proper failure boundaries. Ceph allocates and assigns a process called object storage daemon (OSD) to each disk.
    • A replication factor of 2 is configured, consisting of one original instance plus one replica copy. Ceph ensures that each replica is on a different storage node.
    • A Ceph monitor process (mon) is configured on each storage node.
    • The correct number of placement groups are automatically configured, based on the number of disk drives in the cluster.
    • Properly identified SSD drives are set up for use as Ceph OSD journals to reduce write latencies.
    • An NFS server is created in a virtual machine within the cluster to support virtual machine migration. The NFS file system is mounted on all storage nodes, and every storage node has a shared Nova directory under /var/lib/nova/instances. By default, this NFS file system is configured to utilize 30% of the total initial Contrail storage capacity.

    Using the Contrail Storage User Interface

    The Contrail storage solution provides a user interface integrated into the Contrail user interface. The storage solution user interface displays the following:

    • Customer usable space, which is different from Ceph total space. The displayed usable space does not display the space used by replication and other Ceph functions.
    • Monitor OSDs (disks), monitoring processes (MON), and state changes, enabling quick identification of resource failures within storage components.
    • Total cluster I/O statistics and individual drive statistics.
    • Ceph-specific information about each OSD (disk).
    • Ceph logs, Ceph nodes, and Ceph alerts.

    Use Monitor > Infrastructure > Dashboard to get an “at-a-glance” view of the system infrastructure components, including the numbers of virtual routers, control nodes, analytics nodes, config nodes, and storage nodes currently operational, and a bubble chart of storage nodes showing the Available (%) and Total storage (GB). See the following figure.

    Bubble charts use the following color-coding scheme for storage nodes:

    • Blue—working as configured.
    • Red—error, node is down.
    • Yellow—​one of the node disks is down.

    Select Monitor > Storage > Dashboard to see a summary of cluster health, usage, pools, and disk status, and to gain insight into activity statistics for all nodes. See the following figure.

    Hardware Specifications

    The following are additional hardware specifications needed for the Contrail storage solution.

    Additional minimum specifications:

    • Two​ 500 GB, 7200 RPM drives in the server 4 and server 5 cluster positions (those with the compute storage role) in the Contrail installation. This configuration provides 1 TB of clustered, replicated storage.

    Recommended compute storage configuration:

    • For every 4-5 HDD devices on one compute storage node, use one SSD device to provide the OSD journals for that set of HDD devices.

    Software Files for Compute Storage Nodes

    The Contrail storage solution is only supported with the Ubuntu operating system.

    For each compute storage node, ensure the following software is downloaded:

    • The storage Debian package: contrail-storage-packages_x.xx-xx~xxxxxx_all.deb.
    • NFS VM qcow2 image from Juniper.

    Contrail OpenStack Nova Modifications

    Contrail's OpenStack Nova function has been modified to spawn both migratable and non-migratable virtual machines.

    • Nova's typical virtual machine storage directory, /var/lib/nova/instances, is used for non-migratable virtual machine ephemeral storage.
    • Contrail storage creates a new directory, /var/lib/nova/instances/global, used for the ephemeral storage for migratable virtual machines. The /var/lib/nova/instances/global must be mounted on a shared storage device (NFS with Contrail Storage), accessible from all the compute nodes.
    • To start a non-migratable virtual machine with the Nova CLI command nova boot, the additional argument "--meta storage_scope=local" must be provided.
    • To start a migratable virtual machine with nova boot, the additional argument "--meta storage_scope=global" must be provided. To force Nova and the Horizon UI to spawn migratable virtual machines by default, the storage scope must be set to global. This task is described in the next section.

    Installing the Contrail Storage Solution

    The Contrail storage solution can be installed using the same tools used to install Contrail, either by using Fabric (fab) commands or by using the Contrail Server Manager.

    Both installation methods are described in the following sections.

    Installation Notes

    • When installing a base operating system on any compute storage node, the operating system must be installed only on a single drive. The other drives must be configured as individual devices, and should not be concatenated together in a logical volume manager (LVM) device.
    • For best performance, it is recommended to use solid state devices (SSD) for the Ceph OSD journals. Each SSD device can provide OSD journal support to 3-6 HDD OSD devices, depending on the model of SSD device. Most SSD devices can support up to 4 HDDs, assuming the HDDs are running at capacity.

    Using Fabric Commands to Install and Configure Storage

    Use the information in this section to install storage using Fabric (fab) commands.

    When installing the operating system on a compute storage node, install the operating system on a single drive and leave all other drives as unbundled.

    Installing the Contrail storage solution with Fabric commands gives the following:

    • Base Ceph block device and object support.
    • Easy configuration of SSD devices for OSD journals.
    • Virtual machine migration support.
    • Limited Cinder multi-backend support.

    Cautions

    Before installing, ensure the following:

    • Manually ensure that the UID or GID of the Nova user is identical on all compute nodes before provisioning any software.
    • Manually ensure that the time is identical on all nodes by configuring NTP.

    Fabric Installation Procedure

    This section provides guidelines and steps for using Fabric (fab) commands to install the Contrail storage solution. The installation is similar to a regular Contrail fab installation, however, you will define additional storage information in the testbed.py, including:

    • Define new roles: storage-master and compute-storage.
    • Define how each additional non-root drive is used in the cluster.
    • Define potential additional virtual machine migration variables.
    • Copy and install the additional storage package to systems.
    1. Install the storage Debian package to all nodes:

      fab install_storage_pkg_all:/YYYY/contrail-storage-package-XXX.deb

    2. After issuing fab install_contrail, issue fab install_storage.
    3. After issuing fab setup_all, issue fab setup_storage.
    4. If Contrail-based live virtual machine migration needs to be enabled, issue fab setup_nfs_livem or fab setup_nfs_livem_global, as described in the following

      Note: If virtual machine migration is not needed, do not issue either command.

      • Use fab setup_nfs_livem to store the virtual machine's ephemeral storage on local drives.
      • Use fab setup_nfs_livem_global to store the virtual machine's ephemeral storage within Contrail's storage (using Ceph). This command sets the cluster storage scope to global.
    5. Add two new Contrail storage roles: compute-storage and storage-master.
      • Define the storage-master role on all nodes running OpenStack. Although Ceph has no notion of a master, define this role because Ceph must be run on the node that runs the OpenStack software. OpenStack nodes typically do not have any cluster storage defined, only local storage.
      • The storage-compute role is an add-on role, which means that compute nodes have the option of providing storage functionality. Standalone storage nodes are not supported.
    6. Change the testbed.py details as needed for your environment.

      In the base configuration, define the storage_node_config, which gives device details. See the following example.

      storage_node_config = {
      
      host2 : { 'disks' : ['/dev/sdb', '/dev/sdc'], 'ssd-disks': ['/dev/sdd', '/dev/sde'], 'local-disks' : ['/dev/sdf', '/dev/sdh'], 'nfs' : ['10.87.140.156:/test', '10.87.140.156:/test1']},
      
      host3 : { 'disks' : ['/dev/sdb:/dev/sde', '/dev/sdc:/dev/sde', '/dev/sdd:/dev/sde', '/dev/sdf:/dev/sdj', '/dev/sdg:/dev/sdj', '/dev/sdh:/dev/sdj', '/dev/sdi:/dev/sdj', 'local-ssd-disks' : ['/dev/sdk', '/dev/sdl']},}

      Available device details parameters include:

      • disks and ssd-disks are Ceph disks.
      • local-disk and local-ssd-disks are LVM disks.
      • host2 in the example shows all the storage types that can be configured using Cinder multi-backend.
      • disks is a list of HDD disks used for a Ceph HDD pool.
      • ssd-disks is a list of SSD disks used for a Ceph SSD pool.
      • local-disks is a list of disks used for local LVM storage.
      • nfs is an NFS device.
      • In the example, host3 is a more typical configuration.
      • /dev/sde and /dev/sdj are SSD disks which are used as OSD journals for other HDD drives.
      • local-ssd-disks is a list of disks used for local SSD LVM storage.
    7. Add virtual machine migration as needed, using the following parameters.

      live_migration = True

      ceph_nfs_livevm = True

      ceph_nfs_livem_subnet = ‘<ip address subnet>’ # Private subnet to be provided for live migration VM

      ceph_nfs_livem_image = ‘/Ubuntu/libmnfs.qcow2’ # path of live migration qcow2 image. This image is provided by Juniper.

      ceph_nfs_livem_host = host3 # host in which the NFS VM will run

      For external NFS server based live migration, use the following configuration, valid for Contrail Release 2.20 and greater.

      live_migration = True

      ext_nfs_livevm = True

      ext_nfs_livem_mount = '10.10.10.10:/nfsmount' # External NFS server mount path

      Note: when using an external NFS server, make sure the NFS server maps the uids/gids correctly and provides read/write access for all the uids. If there is any issue related to the permission, either the VM launch will error out or the live migration will fail with permission related errors.

    Using Server Manager to Install and Configure Storage

    This section provides notes and guidelines to install the storage solution using the Contrail Server Manager. Installing the Contrail Storage solution using Server Manager provides:

    • Base Ceph block device and object support.
    • Easy configuration of SSD journals.
    • Support for live migration configuration, starting with Contrail Release 2.10.

    Before installing the base operating system with Server Manager, ensure that the compute storage nodes have been configured with single operating system device installs.

    Cautions

    • Virtual machine migration support uses a fixed IP (192.168.101.3) for the livemnfs virtual machine, starting with Contrail Release 2.10.
    • There is no Cinder multi-backend support.
    • There is no support for single server provisioning, the entire cluster must be provisioned.

    Server Manager Installation Procedure for Storage

    This section provides notes and guidelines if you choose to install the storage solution using the Contrail Server Manager.

    1. Upload the storage package: server-manager add image -f <filename.json>

      where <filename.json> has content similar to the following example:

      {
      
          "image": [
      
             {
      
                  "id": "contrail-storage-packages_1.10-xx~xxxxxx_all",
      
                  "parameters": "{}",
      
                  "path": "/store/contrail-storage-packages_1.10-xx~xxxxxx_all.deb",
      
                  "type": "contrail-storage-ubuntu-package",
      
                  "version": "1.10-xx"
      
              }, 
      
          ]
      
      }
    2. Run ceph-authtool if you need to generate unique keys for administration, monitor, and OSD.
      1. To install ceph-authtool on CentOS:

        yum install http://ceph.com/rpm/el6/x86_64/ceph-common-0.80.5-0.el6.x86_64.rpm

      2. To install ceph-authtool on Ubuntu:

        apt-get install ceph-common

      3. Run this command once for each key:

        ceph-authtool --gen-print-key

      4. Add the generated keys to the cluster.json file:
        cluster.json:
        
        "storage_mon_secret":
        
        "AQBDCdpTsB5FChAAOzI2++uosfmtj7tjmhPu0g==”,
        
        "osd_bootstrap_key":
        
        "AQBKCdpTmN+HGRAAl6rmStq5iYoPnANzSXLcXA==”,
        
        "admin_key":
        
        "AQBLCdpTuOS6FhAAfDW0SsdzyDAUeuwOr/h61A==”
      5. Starting with Release 2.10, add live-migration configuration to the cluster.json file, if you are using live migration.
        "live_migration": "enable",
        
        "live_migration_nfs_vm_host":"compute-node-01",
        
        "live_migration_storage_scope":"global",

    Example Configurations for Storage for Reimaging and Provisioning a Server

    Use the following example configurations as guidelines for reimaging and provisioning a server for storage. Examples are given for configurations for releases prior to Release 2.10 and for configurations for Release 2.10 and greater.

    1. Define storage in the cluster. The following example configurations show new key-value pairs added to the configuration. The “cluster” section should appear similar to the following when storage is defined in a cluster.

      Example: Storage and key-value pairs defined in releases prior to 2.10:

      {
      
          "cluster" : [
      
              { 
      
                  "id" : "demo-cluster",
      
                  "parameters" : {
      
                      "router_asn": "<asn>",
      
                      "database_dir": "/home/cassandra",
      
                      "database_token": "",
      
                      "use_certificates": "False",
      
                      "multi_tenancy": "False",
      
                      "encapsulation_priority": "MPLSoUDP,MPLSoGRE,VXLAN",
      
                      "service_token": "<password>",
      
                      "keystone_user": "admin",
      
                      "keystone_password": "<password>",
      
                      "keystone_tenant": "admin",
      
                      "analytics_data_ttl": "168",
      
                      "subnet_mask": "<ip address>",
      
                      "gateway": "<ip address>",
      
                      "password": "<password>",
      
                      "haproxy": "disable",
      
                      "external_bgp": "",
      
                      "domain": "demo.company.net",
      
      "storage_mon_secret": "$ABC123”,
      
      "osd_bootstrap_key": "$ABC123”,
      
      "admin_key": "$ABC123”
      
      
      
                  }
      
              }
      
          ]
      
      }
      

      Example: Storage and key-value pairs defined in releases 2.10 and greater:

      {
      
          "cluster" : [
      
              { 
      
                  "id" : "demo-cluster",
      
                  "parameters" : {
      
                      "router_asn": "<asn>",
      
                      "database_dir": "/home/cassandra",
      
                      "database_token": "",
      
                      "use_certificates": "False",
      
                      "multi_tenancy": "False",
      
                      "encapsulation_priority": "MPLSoUDP,MPLSoGRE,VXLAN",
      
                      "service_token": "<password>",
      
                      "keystone_user": "admin",
      
                      "keystone_password": "<password>",
      
                      "keystone_tenant": "admin",
      
                      "analytics_data_ttl": "168",
      
                      "subnet_mask": "<ip address>",
      
                      "gateway": "<ip address>",
      
                      "password": "<password>",
      
                      "haproxy": "disable",
      
                      "external_bgp": "",
      
                      "domain": "demo.company.net",
      
      "storage_mon_secret": "$ABC123”,
      
      "osd_bootstrap_key": "$ABC123”,
      
      "admin_key": "$ABC123”,
      
      "live_migration" : "enable",
      
      "live_migration_nfs_vm_host": "compute-host-01,
      
      "live_migration_storage_scope": "global",
      
                  }
      
              }
      
          ]
      
      }
      
      
    2. Add the disks key, the storage-compute role value, and the storage_repo_id.
      • The storage_repo_id key must be added to servers with the storage-master or storage-compute roles.
      • The disks key-value pair must be added to servers with the storage-compute roles.
      • The storage-master value must be added to the "roles" key for the server that has the storage-master role.
      • The storage-compute value must be added to the roles key for the servers which have the storage-compute role.

        The following server section is an example, showing the new keys storage_repo_id and disks, and the new values storage-compute and storage-master. ​

        In the example, one server contains the storage-compute role and has 3 HDD drives (/dev/sdb, /dev/sdc, /dev/sdd​), supporting 3 OSDs.

        Each OSD uses one partition of an SSD drive (/dev/sde) as its OSD journal.

        The server manager software will correctly partition /dev/sdd and assign one partition to each OSD. The storage_repo_id contains the base name of the Contrail storage package which has been added as an image to Server Manager.

      Example: Server.json updates defined in releases prior to 2.10:

      {"server": [
      
              {
      
                  "id": "demo2-server",
      
                  "mac_address": "<mac address>",
      
                  "ip_address": "<ip address>",
      
                  "parameters" : {
      
                      "interface_name": "eth1",
      
                      "compute_non_mgmt_ip": "",
      
                      "compute_non_mgmt_gway": "",
      
                      "storage_repo_id":"contrail-storage-packages",
      
                      "disks": ["/dev/sdb:/dev/sde", "/dev/sdc:/dev/sde", "/dev/sdd:/dev/sde"]​
      
                  },
      
                  "roles" :
      
      ["config","openstack","control","compute","collector","webui","database","storage-compute","storage-master"],
      
                  "cluster_id": "demo-cluster",
      
                  "subnet_mask": "<ip address>",
      
                  "gateway": "<ip address>",
      
                  "password": "<password>",
      
                  "domain": "demo.company.net",
      
                  "email": "id@company.net"
      
      } ]
      
      }
      
      

      Example: Server.json updates defined in releases 2.10 and greater:

       Server.json :
      
      {"server": [
      
              {
      
                  "id": "demo2-server",
      
                  "mac_address": "<mac address>",
      
                  "ip_address": "<ip address>",
      
                  "parameters" : {
      
                      "interface_name": "eth1",
      
                      "compute_non_mgmt_ip": "",
      
                      "compute_non_mgmt_gway": "",
      
                      "storage_repo_id":"contrail-storage-packages",
      
                      "disks": ["/dev/sdb:/dev/sde", "/dev/sdc:/dev/sde", "/dev/sdd:/dev/sde"]​
      
                  },
      
                  "roles" :
      
      ["config","openstack","control","compute","collector","webui","database","storage-compute","storage-master"],
      
                      "contrail": {
      
                          "control_data_interface": "p3p2"
      
                     },
                      "network": {
                      "interfaces": [
                          {
                              "default_gateway": "<ip address>",
                              "dhcp": true,
                              "ip_address": "<ip address>",
                              "mac_address": "<mac address>",
                              "member_interfaces": "",
                              "name": "eth1",
                              "tor": "",
                              "tor_port": "",
                              "type": "physical"
                          },
                          {
                              "default_gateway": "<ip address>",
                              "dhcp": "",
                             "ip_address": "<ip address>",
                              "mac_address": "<mac address>",
                              "member_interfaces": "",
                              "name": "p3p2",
                              "tor": "",
                              "tor_port": "",
                              "type": "physical"
                          }
                      ],
                      "management_interface": "eth1"
                  },
      
                  "cluster_id": "demo-cluster",
      
                  "subnet_mask": "<ip address>",
      
                  "gateway": "<ip address>",
      
                  "password": "<password>",
      
                  "domain": "demo.company.net",
      
                  "email": "id@company.net"
      
      } ]
      
      }
    3. Use the following commands to provision the entire cluster:

      # /opt/contrail/server_manager/client/server-manager -c

      # /opt/contrail/server_manager/smgr_config.ini provision --cluster_id test-cluster contrail_test_pkg

    Storage Installation Limits

    General Limitations

    • Minimum number of storage nodes to configure: 2
    • The number of storage nodes should always be an even number (2, 4, 12, 22, etc.).

    Fab Storage Install Limitations

    There are no limitations to installation when using fab commands.

    Server Manager Storage Install Limitations

    • There is no integrated way to add OSDs or drives to a storage node.
    • There is no integrated way to add new storage nodes to a cluster.
    • Provisioning a single server is not supported. You can add a server to Server Manager and then provision the entire cluster.
    • The live migration overlay network is preset to use 192.168.101.0/24.
    • The user must copy the image livemnfs.qcow2.gz to the folder /var/www/html/contrail/images before provisioning live migration using Server Manager.

    Modified: 2016-06-13