Setup

Networking

We'll use 192.168.8.0/24 for the Ceph public network -- that's the one the clients use to talk to the monitors and storage nodes.

We'll also use 192.168.9.0/24 for the Ceph cluster network. This is entirely optional and is used by the storage nodes to copy blobs between themselves. If it isn't set up they'll use the public network.

ceph.conf

cat <<EOF > /etc/ceph/${CN}.conf
[global]
fsid = ${FSID}
mon initial members = ${HOST}
mon host = ${MON_IP}
public network = 192.168.8.0/24
cluster network = 192.168.9.0/24
auth cluster required = cephx
auth service required = cephx
auth client required = cephx
osd pool default size = 3
osd pool default min size = 2
osd pool default pg num = 128
osd pool default pgp num = 128

# 1 normally, 0 for a single machine hack
osd crush chooseleaf type = 1

[mon.${HOST}]
mon allow pool delete = true
EOF

chown ceph:ceph /etc/ceph/${CN}.conf

Monitor

Again, this looks complicated but it's mostly about creating ACLs for various purposes.

Keyrings bind a name (-n *name*) with a set of capabilities (--cap *thing* *ACL*).

• Create a temporary keyring for the monitors to use -- the monitors only need to see the, er, monitors:

  sudo -u ceph ceph-authtool --create-keyring /tmp/ceph.mon.keyring --gen-key -n mon. --cap mon 'allow *'
  

  Note

  The name mon. is used to retrieve the monitor key when we add more monitors.

• Create a permanent keyring for admins (us!) to use -- we need access to everything:

  sudo ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *'
  

• Create a permanent keyring for bootstrapping OSDs:

  sudo ceph-authtool --create-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring --gen-key -n client.bootstrap-osd --cap mon 'profile bootstrap-osd'
  

• Add the second two permanent keyrings (admins, bootstrap OSDs) to the first temporary keyring (monitors):

  sudo ceph-authtool /tmp/ceph.mon.keyring --import-keyring /etc/ceph/ceph.client.admin.keyring
  sudo ceph-authtool /tmp/ceph.mon.keyring --import-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring
  

• Create the initial monitor map with this host in it using the temporary keyring:

  monmaptool --create --add ${HOST} ${MON_IP} --fsid {$FSID} /tmp/monmap
  

• Create the working directory for the monitor daemon on this host:

  sudo -u ceph mkdir /var/lib/ceph/mon/${CN}-${HOST}
  

• Start the monitor with the initial monitor map:

  sudo -u ceph ceph-mon --mkfs -i ${HOST} --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring
  

  Note

  The monitor is now running. We've just run the above command in the foreground and, given that nothing else is running in our cluster, we can ^C it and restart it with systemd in two ticks.

• The instructions say "create the done file" so, er, we do it:

  sudo -u ceph touch /var/lib/ceph/mon/${CN}-${HOST}/done
  

• Finally, kick off the monitor daemon on this host:

  sudo systemctl enable ceph-mon@${HOST}
  sudo systemctl start ceph-mon@${HOST}
  

That's it, our monitor daemon is now live. Remember, the monitor map is live. That initial monitor map we created remains just that, a point in time instance of the map. If we rebooted now the monitor would revert to that old map. Which might not be ideal.

How do we know it is running?

# ceph -s

You should get something even it reports some errors about the cluster state. We only have a monitor node so it shouldn't be too surprising that the cluster is in ill-health.

You can also run a long term variant of the above:

ceph -w

which prints out the same initial status then hangs about printing any state changes. Most of the time nothing much happens other than Ceph complaining that time is out of sync (for reasons unknown as these systems are generally sub-millisecond in sync). However, if you restart an OSD you'll get some more interesting messages.

Firewall

Just to tidy things up, open up the firewall to allow other nodes to talk to us:

# port 6789
firewall-cmd --add-service=ceph-mon --permanent

In addition (beyond the official docs) I believe you need to open up the general Ceph ports (6800-7300) as the monitors maintain connections to all the storage nodes:

firewall-cmd --add-service=ceph --permanent

Firewall

# mgr!
firewall-cmd --add-port=6810/tcp --permanent

Dashboard

The Ceph Manager supports a decent Web UI dashboard:

ceph mgr module enable dashboard

(you only need to do this once per cluster)

and open the firewall on each manager node:

# mgr dashboard
firewall-cmd --add-port=8443/tcp --permanent

One nice aspect is that you can bookmark any of the monitor/manager nodes in your browser and you will be automatically redirected to the current live one.

Other Monitors

Other monitors are a little more involved as the cluster map is live. We need to get a copy of the live map, add our new monitor and then kick off the daemon.

Having installed the software and copied the config as above:

• Log into the new monitor and prep the working directory:

  ssh mon2
  HOSTNAME=$(uname -n)
  HOST=${HOSTNAME%%.*}
  
  mkdir /var/lib/ceph/mon/ceph-${HOST}
  

• Collect the mon. key and the current cluster map:

  ceph auth get mon. -o key
  ceph mon getmap -o map
  

  The mon. key was the name used for all monitors.

• Kick off the new monitor using the key and map we just saved:

  ceph-mon -i ${HOST} --mkfs --monmap map --keyring key
  

  Note

  The cluster map is live so the act of this new monitor starting (with the right credentials and latest map) means the cluster immediately has an extra monitor. Nothing more to do here.

• Ready this system to start the monitor on reboot:

  chown -R ceph:ceph /var/lib/ceph/mon/ceph-${HOST}
  systemctl enable ceph-mon@${HOST}
  

ceph osd tree

At any point you can run ceph osd tree to see your accumulated set of OSDs:

# ceph osd tree
ID  CLASS WEIGHT  TYPE NAME     STATUS REWEIGHT PRI-AFF
 -1       9.70517 root default
 -2       1.54099     host c1c
  3   hdd 0.77049         osd.3     up  1.00000 1.00000
  2   ssd 0.77049         osd.2     up  1.00000 1.00000
-10       1.81839     host c2c
  4   hdd 0.90919         osd.4     up  1.00000 1.00000
  5   ssd 0.90919         osd.5     up  1.00000 1.00000
 -5       6.34579     host c3c
  0   hdd 3.17290         osd.0     up  1.00000 1.00000
  1   hdd 3.17290         osd.1     up  1.00000 1.00000

Here I don't have the full 1TB/4TB (essentially the weight column) as I've chewed off disk partitions for an OpenStack Swift implementation.

OSD Class

Note that Ceph has recognised SSDs and HDDs and designated the appropriate class. It doesn't differentiate NVMes (at the time of writing) and marks them as SSDs. You can revisit this if you desire with something like:

ceph osd crush rm-device-class ${osd_id}
ceph osd crush set-device-class nvme ${osd_id}

Does the class do anything? Well, not immediately (or at least not immediately obviously). The device class is used (or at least available to be used) in placement decision making. We can force it to be used for particular pools.

OSD removal

You can "cleanly" remove an OSD if you decide to repurpose your disk. Suppose we want to remove sdb1 then first figure out which OSD is using sdb1:

ceph-volume lvm list

Now you can mark the corresponding osd.N as down:

systemctl stop ceph-osd@N

and now actually zap the disk:

ceph-volume lvm zap /dev/sdb1

which may get so far then fail. It's not figured out the volume groups. You might want to make some appropriate decision about who's using what then manually remove the volume group (and contained logical volume) and re-run ceph-volume lvm zap. Something like:

pvs /dev/sdb1

to figure out the volume group used by sdb1 then:

vgremove -y $UUID

and finally repeat:

ceph-volume lvm zap /dev/sdb1

What we're aiming for is to have the OSD ID freed up. The ultimate test of the success of that is that a subsequent ceph-volume lvm create will re-use that freed up OSD ID.

If it didn't then you may need to remove references to the OSD ID from various places:

ceph osd rm osd.$ID
ceph osd crush rm osd.$ID
ceph auth rm osd.$ID

OSD Usage

You can get a view of how your OSDs are being used, something like:

# ceph osd df
ID CLASS WEIGHT  REWEIGHT SIZE    RAW USE DATA    OMAP    META     AVAIL   %USE  VAR  PGS STATUS
 3   hdd 0.77049  1.00000 789 GiB 375 GiB 294 GiB 100 KiB 1024 MiB 414 GiB 47.55 2.78 102     up
 2   ssd 0.77049  1.00000 789 GiB 130 GiB  49 GiB  20 KiB 1024 MiB 659 GiB 16.52 0.97  90     up
 4   hdd 0.90919  1.00000 931 GiB 289 GiB 288 GiB  28 KiB  1.2 GiB 642 GiB 31.06 1.82  97     up
 5   ssd 0.90919  1.00000 931 GiB  57 GiB  56 GiB  24 KiB 1024 MiB 874 GiB  6.07 0.36  95     up
 0   hdd 3.17290  1.00000 3.2 TiB 406 GiB 155 GiB 176 KiB 1024 MiB 2.8 TiB 12.50 0.73  90     up
 1   hdd 3.17290  1.00000 3.2 TiB 440 GiB 189 GiB  96 KiB 1024 MiB 2.7 TiB 13.53 0.79 102     up
                    TOTAL 9.7 TiB 1.7 TiB 1.0 TiB 445 KiB  6.2 GiB 8.0 TiB 17.08

Although I'm not sure you can really tell much from that as a user. Notice that the larger disks (IDs 0 and 1) don't necessarily have much more data on them. We're back to that balancing across failure points (hosts) issue.

Pool Creation

Pool creation is easy enough, barring this unclear decision about placement groups:

ceph osd pool create glance-images 64

where 64 is the number of placement groups to use. If you decide you're unhappy with something about your pool you can fiddle with its parameters:

ceph osd pool cinder-volumes [get|set] parameter [...]

If you omit the parameter you'll get a list of possible parameters.

At this point we can do a simple list of the pools:

# ceph osd lspools
1 glance-images
2 cinder-volumes
3 nova-vms

although a rados df is more illuminating about the overall state of play (rados is a command for interacting with a Ceph object storage cluster):

# rados df
POOL_NAME         USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED    RD_OPS      RD    WR_OPS      WR USED COMPR UNDER COMPR
cinder-volumes  13 GiB    1134      0   3402                  0       0        0  55083257  49 GiB   3440171 119 GiB        0 B         0 B
glance-images  706 GiB   30019      0  90057                  0       0        0  54244420 230 GiB    238858 329 GiB        0 B         0 B
nova-vms       315 GiB   15452      0  46356                  0       0        0 384551762 676 GiB 385853274  13 TiB        0 B         0 B

total_objects    46605
total_used       1.7 TiB
total_avail      8.0 TiB
total_space      9.7 TiB

Remember that the usage is going to reflect your replication ratio.

Pool Destruction

Getting rid of pools isn't quite so easy. We need to be allowed to do it and we need to express our determination to do so.

Firstly, we need to be allowed to delete a pool at all. You'll need to edit /etc/ceph/ceph.conf:

[mon.mon3]
mon allow pool delete = true

and then restart the daemon:

systemctl restart ceph-mon@mon3

The actual delete command requires extra effort too:

# ceph osd pool delete foo
Error EPERM: WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo.  If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it.

OK:

# ceph osd pool delete foo foo --yes-i-really-really-mean-it
pool 'foo' removed

which is trivially scripted.