osd is not the same as ‘crush osd’

I still can not get used to the idea that OSD map is not the same thing as crush map with OSD’s.

It is a funny bug/feature I found with option ‘osd crush update on start = False’ in [osd] section in ceph.conf (or may be even without it, I’m not sure).

When new OSD is added manually (no ceph-disk used) it visible in ceph osd tree, but unavailable in crush manipulation. I wrote about one that case before, but now I think I understand what’s going on.

Main symptom: you can see osd in ‘ceph osd tree’ in up state outside of any tree, but when you’re trying to move it you’re getting an error:

ceph osd crush set osd.0 1.0 host=lab0
Error ENOENT: unable to set item id 0 name ‘osd.0’ weight 1.00 at location {host=lab0}: does not exist

You need to add it to the crush map manually. Not only ceph osd add’, but also ‘ceph osd crush add’ with same syntax as ‘ceph osd crush set’:

ceph osd crush add osd.0 1.0 host=lab0
add item id 0 name 'osd.0' weight 1.0 at location {host=lab0} to crush map

You can see all ‘crush osds’ by querying ceph itself: ceph osd crush tree (do not mix with ceph osd tree).

Like what you read? Give George Shuklin a round of applause.

From a quick cheer to a standing ovation, clap to show how much you enjoyed this story.