B.4. Region Overlap Repairs
Table integrity problems can require repairs that deal with overlaps. This is a riskier operation
because it requires modifications to the file system, requires some decision making, and may
require some manual steps. For these repairs it is best to analyze the output of a
run so that you isolate repairs attempts only upon problems the checks identify. Because this is
riskier, there are safeguard that should be used to limit the scope of the repairs.
WARNING: This is a relatively new and have only been tested on online but idle HBase instances
(no reads/writes). Use at your own risk in an active production environment!
The options for repairing table integrity violations include:
-fixHdfsOrphans option for “adopting” a region directory that is missing a region
metadata file (the .regioninfo file).
-fixHdfsOverlaps ability for fixing overlapping regions
When repairing overlapping regions, a region’s data can be modified on the file system in two
ways: 1) by merging regions into a larger region or 2) by sidelining regions by moving data to
“sideline” directory where data could be restored later. Merging a large number of regions is
technically correct but could result in an extremely large region that requires series of costly
compactions and splitting operations. In these cases, it is probably better to sideline the regions
that overlap with the most other regions (likely the largest ranges) so that merges can happen on
a more reasonable scale. Since these sidelined regions are already laid out in HBase’s native
directory and HFile format, they can be restored by using HBase’s bulk load mechanism.
The default safeguard thresholds are conservative. These options let you override the default
thresholds and to enable the large region sidelining feature.
-maxMerge <n> maximum number of overlapping regions to merge
-sidelineBigOverlaps if more than maxMerge regions are overlapping, sideline attempt
to sideline the regions overlapping with the most other regions.
-maxOverlapsToSideline <n> if sidelining large overlapping regions, sideline at most n
Since often times you would just want to get the tables repaired, you can use this option to turn
on all repair options:
-repair includes all the region consistency options and only the hole repairing table
Finally, there are safeguards to limit repairs to only specific tables. For example the following
command would only attempt to check and repair table TableFoo and TableBar.
$ ./bin/hbase hbck -repair TableFoo TableBar
B.4.1. Special cases: Meta is not properly assigned
There are a few special cases that hbck can handle as well.
Sometimes the meta table’s only region is inconsistently assigned or deployed. In this case
there is a special
option that can try to fix meta assignments.
$ ./bin/hbase hbck -fixMetaOnly -fixAssignments
B.4.2. Special cases: HBase version file is missing
HBase’s data on the file system requires a version file in order to start. If this flie is missing, you
can use the
option to fabricating a new HBase version file. This assumes that
the version of hbck you are running is the appropriate version for the HBase cluster.
B.4.3. Special case: Root and META are corrupt.
The most drastic corruption scenario is the case where the ROOT or META is corrupted and
HBase will not start. In this case you can use the OfflineMetaRepair tool create new ROOT
and META regions and tables.
This tool assumes that HBase is offline. It then marches through the existing HBase home
directory, loads as much information from region metadata files (.regioninfo files) as possible
from the file system. If the region metadata has proper table integrity, it sidelines the original root
and meta table directories, and builds new ones with pointers to the region directories and their
$ ./bin/hbase org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair
NOTE: This tool is not as clever as uberhbck but can be used to bootstrap repairs that uberhbck
If the tool succeeds you should be able to start hbase and run online repairs if necessary.
B.4.4. Special cases: Offline split parent
Once a region is split, the offline parent will be cleaned up automatically. Sometimes, daughter regions
are split again before their parents are cleaned up. HBase can clean up parents in the right order. However,
there could be some lingering offline split parents sometimes. They are in META, in HDFS, and not deployed.
But HBase can't clean them up. In this case, you can use the
-fixSplitParents option to reset
them in META to be online and not split. Therefore, hbck can merge them with other regions if fixing
overlapping regions option is used.
This option should not normally be used, and it is not in