Effortless deployment and scaling of a Ceph cluster

12/02/2013 - Gérard Dethier (@gdethier)

In this blog post, we explain how to deploy and scale a cluster hosting a distributed object store and file system called Ceph in the Cloud using ComodIT’s orchestration possibilities. Orchestration was already illustrated in a previous post with a web cluster hosting a Wordpress blog. We successfully tested the deployment of Ceph cluster on Amazon EC2 and Openstack, however it should be possible to use any platform supported by ComodIT (Eucalyptus, Rackspace, etc.) or even deploy the cluster on a bare-metal infrastructure.

The ceph orchestration scripts are available in the ComodIT repository. These scripts target CentOS distributions.

If you have other orchestration use cases, we’d be happy to help you out. Contact us and let’s discuss about it.

Ceph clusters

A typical Ceph cluster is composed of several instances of each of these services:

• monitors (MONs) which manage the cluster,
• meta-data services (MDSs) which manage the file system namespace,
• object storage services (OSDs) which actually store the data.

In particular, OSDs should be particularly numerous to obtain a large storage. Multiple MONs and MDSs allow to achieve scalability and reliability. Finally, data are replicated among OSDs for reliability.

A Ceph cluster can be scaled in 2 ways:

• increase the number of MONs,
• increase the number of OSDs.

Requirements

1. A ComodIT account (you can register for free here).

2. A ComodIT platform.

   If you havn't done it yet, add your cloud platform to ComodIT. In the ‘Platform’ section, click the Add button, pick a driver and fill in the specific details. For more information, you can follow one of these tutorials.

   Note that you won’t be able to deploy this example on the ‘Demo Platform’ provided by default, which is limited to maximum one provisioning at a time per user.

3. A ComodIT distribution.

   The application templates used in this orchestration scripts have been written for CentOS 6.3. You should therefore deploy on a compatible distribution. If you deploy on EC2, the easier is to get the ‘CentOS 6.3 (AMI)’ image from the ComodIT marketplace.

4. the ComodIT Python library (bundled with command-line interface, see this tutorial for information about how to install it).

Deployment

1. Clone the demos public repository and enter Ceph cluster’s folder:

    git clone git@github.com:comodit/demos.git
    cd demos/ceph-cluster
   

2. Create a config.py file with the following content:

    endpoint = "https://my.comodit.com/api"
   
    username = "<user>"
    password = "<password>"
    organization = "<org_name>"
   
    time_out = 60 * 30  # seconds
   
    admin_key = "AQAEKwlRgBqsDhAA7cwN/JtEyCym6vYN/ixHqA=="
   
    platform = {"name" : "<plat_name>",
                "settings" : { ... }
               }
   
    distribution = {"name" : "<dist_name>",
                    "settings" : { ... }
                   }
   

   where <user> and <password> are your ComodIT credentials, <org_name> the name of your organization, <plat_name> the name of a platform in your organization and <dist_name> the name of a distribution in your organization. You should also fill the settings for both platform and distribution.

   Warning: The applications you will deploy in next step are only compatible with CentOS 6.3! Therefore choose your ComodIT distribution accordingly. Distribution from the store like ‘CentOS 6’ and ‘CentOS 6.3 AMI’ can be used.

   For instance, you may use an Amazon EC2 platform and store’s CentOS 6.3 AMI. In this case, platform settings look like:

    "settings" : {
                 "ec2.instanceType": "t1.micro",
                 "ec2.securityGroups": "default",
                 "ec2.zone": "eu-west-1a",
                 "ec2.keyPair": "<key name>"
                 }
   

   where <key name> is a key pair name and distribution takes no setting:

    "settings" : {}
   

3. Setup you ComodIT account i.e. create all required applications and create an environment that will contain cluster’s hosts:

     ./setup.py
   

4. Actually deploy the cluster:

    ./deploy.py
   

   A simple Ceph cluster composed of 1 MON, 1 MDS and 2 OSDs hosted by 3 hosts is deployed: the MON and the MDS are hosted by the same host, the OSDs have their own host. Of course, this is not an architecture to use in production, you should always have several MONs. The complete deployment takes a few minutes on Amazon EC2.

5. Deployment script prints the public address of what we call the master node i.e. the computer hosting the monitor and MDS. You can connect to this host using SSH and check cluster’s health using the following command (executed as super-user or root):

     ceph -s
   

   See Ceph’s documentation for more details.

Scaling up (OSDs)

Add an OSD to deployed cluster:

./scale_osds.py -c 1

-c option is the number of OSDs to add.

Scaling up (MONs)

Add a monitor to deployed cluster:

./scale_mons.py -c 1

-c option is the number of monitors to add.

Shutting down cluster

You can delete all hosts created during deployment and scaling operations:

./teardown.py

If you also want to clean-up your organization i.e. delete the applications and environment created by setup script:

./cleanup.py