B.3. Localized repairs

When repairing a corrupted HBase, it is best to repair the lowest risk inconsistencies first. These are generally region consistency repairs -- localized single region repairs, that only modify in-memory data, ephemeral zookeeper data, or patch holes in the META table. Region consistency requires that the HBase instance has the state of the region’s data in HDFS (.regioninfo files), the region’s row in the hbase:meta table., and region’s deployment/assignments on region servers and the master in accordance. Options for repairing region consistency include:

To fix deployment and assignment problems you can run this command:

$ ./bin/hbase hbck -fixAssignments

To fix deployment and assignment problems as well as repairing incorrect meta rows you can run this command:

$ ./bin/hbase hbck -fixAssignments -fixMeta

There are a few classes of table integrity problems that are low risk repairs. The first two are degenerate (startkey == endkey) regions and backwards regions (startkey > endkey). These are automatically handled by sidelining the data to a temporary directory (/hbck/xxxx). The third low-risk class is hdfs region holes. This can be repaired by using the:

$ ./bin/hbase hbck -fixAssignments -fixMeta -fixHdfsHoles

Since this is a common operation, we’ve added a the -repairHoles flag that is equivalent to the previous command:

$ ./bin/hbase hbck -repairHoles

If inconsistencies still remain after these steps, you most likely have table integrity problems related to orphaned or overlapping regions.

comments powered by Disqus