Apache Kudu series: 3. Troubleshooting:- Add or remove data directories.

This article covers the steps required to add/remove the data directory to/from a Kudu tablet server.

kudu fs update_dirs tool -

Users can add or remove data directories to an existing master or tablet server via the kudu fs update_dirs tool.
Data is striped across data directories, and when a new data directory is added, new data will be striped across the union of the old and new directories.

How to remove a bad disk as a data directory to a Kudu tablet server?

1. The tablet server should be offline. You can stop this from Cloudera Manager.

2. Run the tool with the desired directory configuration flags.

e.g. If a cluster was set up with — fs_wal_dir=/wals, — fs_metadata_dir=/meta, and — fs_data_dirs=/data/1,/data/2,/data/3, and /data/3 is to be removed (e.g. due to a disk error), run the command:

$ sudo -u kudu kudu fs update_dirs — force — fs_wal_dir=/wals — fs_metadata_dir=/meta — fs_data_dirs=/data/1,/data/2

3. If using CM, make sure to only update the configurations of the updated server, rather than of the entire Kudu service.

4. Once complete, You cand Start the Kudu Tablet server form Cloudera Manager.

5. Run ksck check to verify cluster health.

$ sudo -u kudu kudu cluster ksck master-01.example.com,master-02.example.com,master-03.example.com

How to add a repaired disk as a data directory to a Kudu tablet server?

When the disk is repaired, remounted, and ready to be reused by Kudu, take the following steps:

1. Make sure that the Kudu portion of the disk is completely empty.

2. Stop the tablet server from CM.

3. Update the CM configuration with the values of the fs_data_dirs flags for the updated sever.

4. Run the update_dirs tool. For example, to add /data/3, run the following:

$ sudo -u kudu kudu fs update_dirs — force — fs_wal_dir=/wals — fs_data_dirs=/data/1,/data/2,/data/3

5. Start the tablet server.

6. Run ksck to verify cluster health.

$ sudo -u kudu kudu cluster ksck master-01.example.com,master-02.example.com,master-03.example.com

If you have missed other part of the series then you can check here,

1. Apache Kudu series: 1. The beginning.
2. Apache Kudu series: 2. Troubleshooting for TABLET_DATA_TOMBSTONED