Apache Kudu series: 2. Troubleshooting for TABLET_DATA_TOMBSTONED

This article covers the topics which are encountered while dealing with large Kudu systems.

How to run a Kudu filesystem check?

Usage:
kudu cluster ksck <master_addresses>

==================
Errors:
==================
Corruption: table consistency check error: 1 out of 55 table(s) are not healthy
FAILED

If you check the Summary by the table, it looks like below.

Summary by table

Scenario 1:-
Below tables are difficult to retrieve back as data dirs may have been removed. In this scenario it is sad, but you may have to remove this table from the kudu filesystem.

Tablet f8891f1b94e64199960cae5de302aa26 of table ‘impala::unit.customlocal_1540559635747_imp_track’ is unavailable: 1 replica(s) not RUNNING
 331b70236e6e48b1b0882e5625ebaaac (mac7.c.my-project-laca.internal:7050): not running [LEADER]
 State: FAILED
 Data state: TABLET_DATA_READY
 Last status: Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet f8891f1b94e64199960cae5de302aa26
All reported replicas are:
 A = 331b70236e6e48b1b0882e5625ebaaac

Scenario 2:-
If the tablet servers hosting a majority of the replicas
are down (i.e. ones reported as “TS unavailable” by ksck), they should be recovered instead if possible.

Tablet bc80fe51b0ae4b3fa18625883987482e of table ‘impala::impala::mystage.dev_vact_x1’ is unavailable: 2 replica(s) not RUNNING
 51baf90e259d4e6d8dc842d3eb6d7a82 (mac1.c.my-project-laca.internal:7050): RUNNING
 e94aac3c56de4e22a08e990561cedf63: TS unavailable [LEADER]
 f6b4a3f6628146e1bb6abf65faf34da6: TS unavailable
 d7dfbde0ab694688a6490445842f44d4 (mac4.c.my-project-laca.internal:7050): not running [NONVOTER]
 State: INITIALIZED
 Data state: TABLET_DATA_TOMBSTONED
 Last status: Tombstoned
 All reported replicas are:
 A = 51baf90e259d4e6d8dc842d3eb6d7a82
 B = e94aac3c56de4e22a08e990561cedf63
 C = f6b4a3f6628146e1bb6abf65faf34da6
 D = d7dfbde0ab694688a6490445842f44d4
 The consensus matrix is:
 Config source | Replicas | Current term | Config index | Committed?
 — — — — — — — -+ — — — — — — — — — — — — + — — — — — — — + — — — — — — — + — — — — — — 
 master | A B* C D~ | | | Yes
 A | A B C D~ | 94 | 2059805 | Yes
 B | [config not available] | | |
 C | [config not available] | | |
 D | A B C | 94 | 2059804 | Yes

How to bring a tablet that has lost a majority of replicas back online?

1. Run a ksck against a tablet.

sudo -u kudu kudu cluster ksck — tablets=bc80fe51b0ae4b3fa18625883987482e mac11.c.my-project-laca.internal,mac12.c.my-project-laca.internal,mac13.c.my-project-laca.internal
Tablet Summary
Tablet bc80fe51b0ae4b3fa18625883987482e of table 'impala::mystage.dev_vact_x1' is unavailable: 2 replica(s) not RUNNING
  51baf90e259d4e6d8dc842d3eb6d7a82 (mac1.c.my-project-laca.internal:7050): RUNNING
  e94aac3c56de4e22a08e990561cedf63  : TS unavailable [LEADER]
  f6b4a3f6628146e1bb6abf65faf34da6: TS unavailable
  d7dfbde0ab694688a6490445842f44d4 (mac4.c.my-project-laca.internal:7050): not running [NONVOTER]
    State:       INITIALIZED
    Data state:  TABLET_DATA_TOMBSTONED
    Last status: Tombstoned

This output shows that, for tablet bc80fe51b0ae4b3fa18625883987482e, the two tablet replicas showingTS unavailable failed. The remaining replica is not the leader, so the leader replica failed as well. This means the chance of data loss is higher since the remaining replica onmac1.c.my-project-laca.internal may have been lagging. In the case of 1 out of 3 surviving replicas, there will be only one healthy replica, so the consensus configuration needs to rewrite which includes only the healthy replica.

Collect healthy data from the last command:

<tablet-id> :- bc80fe51b0ae4b3fa18625883987482e

<Healthy tablet server -uuid>:- 51baf90e259d4e6d8dc842d3eb6d7a82

<Healthy tablet server>:- mac1.c.my-project-laca.internal:7050

2. Run remote_replica unsafe_change_config

e.g. sudo -u kudu kudu remote_replica unsafe_change_config <Healthy tablet server:7150> <tablet-id> <Healthy tablet server -uuid>

sudo -u kudu kudu remote_replica unsafe_change_config mac1.c.my-project-laca.internal:7050 bc80fe51b0ae4b3fa18625883987482e 51baf90e259d4e6d8dc842d3eb6d7a82

Once you run above command, the healthy replicas’ consensus configurations have been forced to exclude the unhealthy replicas and the healthy replicas will be able to elect a leader.

1. The tablet will become available for writes, though it will still be under-replicated.
2. Once the tablet becomes available, the leader tablet master will notice that it is under-replicated, and will cause the tablet to re-replicate until the proper replication factor is restored.

At this point, If you run ksck on tablet, you may see output like below,

 State: INITIALIZED
 Data state: TABLET_DATA_COPYING
 Last status: Tablet Copy: Downloading block 0000000146531905 (13756/18705)

The unhealthy replicas will be tombstoned by the master, causing their remaining data to be deleted.

And few minutes later, you may see ksck output as healthy.

Healthy ksck output.

If you have missed other parts of the series then you can check here,

1. Apache Kudu series: 1. The beginning.
2. Apache Kudu series: 3. Troubleshooting:- Add or remove data directories.