A very slow pool removal

It’s still a problem under investigation, but I want to write it down now.

I found that removing pools after some serious RGW exercise (60M objects in one bucket!) does not freeing the space. To be precise, it does, but at a crawling speed. To cleanup the cluster it took more time than to fill it up.

Moreover, I found that some OMAP’s are not cleaned at all (32GiB of OMAPs for cluster with no pools).

Even now (16 hours since last pool was deleted) I can see that there are about 25 PG left in a cluster with no pools whatsoever, and ‘DATA’ is ceph osd df still around 500Gb. Additionally, I see some high CPU usage for OSD nodes. As far as I can guess, it’s deep scrub killing objects and PGs as they found to be obsolete.

The most concerning part here for me is lack of visibility. This is very laboratory grade cluster with no actual load, so I can use ‘totals’ to guess the progress. If I have had the actual load, I would be completely blind on this, and I don’t like it.

Insofar I found only one visible marker: ceph pg dump_pools_json shows more pools than ceph osd pool ls. Currently I’m playing with ceph daemon osd.0 dump_pgstate_history where I can see some peculiar line “state”: “Started/ToDelete/WaitDeleteReseved”

Nevertheless I feel that OSD daemon for bluestore is a rather poor replacement for simple ‘cd/df’ for filestore.