This is a post that I have had in draft mode for quite some time. At this point some of this information is out of date, so I am planning on writing a ‘part II’ post shortly, which will include some updated information.

Benchmarking Ceph:

Ever since we got our ceph cluster up and running, I’ve been running various benchmarking applications against different cluster configurations. Just to review, the cluster that we recently built has the following specs:

Cluster specs:

• 3 x Dell R-420;32 GB of RAM; for MON/RADOSGW/MDS nodes
• 6 x Dell R-720xd;64 GB of RAM; for OSD nodes
• 72 x 4TB SAS drives as OSD’s
• 2 x Force10 S4810 switches
• 4 x 10 GigE LCAP bonded Intel cards
• Ubuntu 12.04 (AMD64)
• Ceph 0.72.1 (emperor)
• 2400 placement groups
• 261TB of usable space

The main role for this cluster will be one primarily tied to archiving audio and video assets. This being the case, we decided to try and maximize total cluster capacity (4TB drives, no ssd’s, etc), while at the same time being able to achieve and maintain reasonable cluster throughput (10 GigE, 12 drives per osd nodes, etc).

Most of my benchmarking focused on rbd and radosgw, because either of these is most likely to be what we introduce into production when we are ready.  We are very much awaiting a stable and supported cephfs release (which will hopefully be available sometime in mid-late 2014), which will allow us to switch out our rbd + samba setup, for on based on cephfs.

Rados Benchmarks: 

I setup a pool called ‘test’ with 1600 pg’s in order to run some benchmarks using the ‘rados bench’ tool that came with Ceph.  I started with a replication level of ‘1’ and worked my way up to a replication level of ‘3’.

RBD Benchmarks:

Next I setup a 10GB block device using rbd:

Radosgw Benchmarks:

Using s3cmd (s3tools) I was able to achieve about 70MB/s when pushing files to ceph via the s3 restful API.

After spending about 4 months testing, benchmarking, setting up and breaking down various Ceph clusters, I though I would spend time documenting some of the things I have learned while setting up cephfs, rbd and radosgw along the way.

First let me talk a little bit about the details of the cluster that we will be putting into production over the next several weeks.

Cluster specs:

• 6 x Dell R-720xd;64 GB of RAM; for OSD nodes
• 72 x 4TB SAS drives as OSD’s
• 3 x Dell R-420;32 GB of RAM; for MON/RADOSGW/MDS nodes
• 2 x Force10 S4810 switches
• 4 x 10 GigE LCAP bonded Intel cards
• Ubuntu 12.04 (AMD64)
• Ceph 0.72.1 (emperor)
• 2400 placement groups
• 261TB of usable space

The process I used to set- up and tear down our cluster during testing was quite simple, after installing ‘ceph-deploy’ on the admin node:

1. ceph-deploy new mon1 mon2 mon3
2. ceph-deploy install  mon1 mon2 mon3 osd1 osd2 osd3 osd4 osd5 osd6
3. ceph-deploy mon create mon1 mon2 mon3
4. ceph-deploy gatherkeys mon1
5. ceph-deploy osd create osd1:sdb
6. ceph-deploy osd create osd1:sdc
   ……….

The uninstall process went something like this:

1. ceph-deploy disk zap osd1:sdb
   ……….
2. ceph-deploy purge mon1 mon2 mon3 osd1 osd2 osd3 osd4 osd5 osd6
3. ceph-deploy purgedata mon1 mon2 mon3 osd1 osd2 osd3 osd4 osd5 osd6

Additions to ceph.conf:

Since we wanted to configure an appropriate journal size for our 10GigE network, mount xfs with appropriate options and configure radosgw, we added the following to our ceph.conf (after ‘ceph-deploy new but before ‘ceph-deploy install’:

[global]
osd_journal_size = 10240
osd_mount_options_xfs = “rw,noatime,nodiratime,logbsize=256k,logbufs=8,inode64”
osd_mkfs_options_xfs = “-f -i size=2048”

[client.radosgw.gateway]
host = mon1
keyring = /etc/ceph/keyring.radosgw.gateway
rgw_socket_path = /tmp/radosgw.sock
log_file = /var/log/ceph/radosgw.log
admin_socket = /var/run/ceph/radosgw.asok
rgw_dns_name = yourdomain.com
debug rgw = 20
rgw print continue = true
rgw should log = true
rgw enable usage log = true

Benchmarking:

I used the following commands to benchmark rados, rbd, cephfs, etc

1. rados -p rbd  bench 20 write –no-cleanup
2. rados -p rbd  bench 20 seq
3. dd bs=1M count=512 if=/dev/zero of=test conv=fdatasync
4. dd bs=4M count=512 if=/dev/zero of=test conv=fdatasync

 Ceph blogs worth reading:

http://ceph.com/community/blog/
http://www.sebastien-han.fr/blog/
http://dachary.org/