org.apache.hadoop.hbase.util.hbck

Class HFileCorruptionChecker

java.lang.Object

org.apache.hadoop.hbase.util.hbck.HFileCorruptionChecker

@InterfaceAudience.Private
public class HFileCorruptionChecker
extends Object

This class marches through all of the region's hfiles and verifies that they are all valid files. One just needs to instantiate the class, use checkTables(List<Path>) and then retrieve the corrupted hfiles (and quarantined files if in quarantining mode) The implementation currently parallelizes at the regionDir level.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
private class	HFileCorruptionChecker.MobRegionDirChecker An individual work item for parallelized mob dir processing.
private class	HFileCorruptionChecker.RegionDirChecker An individual work item for parallelized regiondir processing.

Field Summary

Fields

Modifier and Type	Field and Description
(package private) CacheConfig	cacheConf
(package private) org.apache.hadoop.conf.Configuration	conf
(package private) Set<org.apache.hadoop.fs.Path>	corrupted
(package private) Set<org.apache.hadoop.fs.Path>	corruptedMobFiles
(package private) ExecutorService	executor
(package private) Set<org.apache.hadoop.fs.Path>	failureMobFiles
(package private) Set<org.apache.hadoop.fs.Path>	failures
(package private) org.apache.hadoop.fs.FileSystem	fs
(package private) AtomicInteger	hfilesChecked
(package private) boolean	inQuarantineMode
private static org.slf4j.Logger	LOG
(package private) Set<org.apache.hadoop.fs.Path>	missedMobFiles
(package private) Set<org.apache.hadoop.fs.Path>	missing
(package private) AtomicInteger	mobFilesChecked
(package private) Set<org.apache.hadoop.fs.Path>	quarantined
(package private) Set<org.apache.hadoop.fs.Path>	quarantinedMobFiles

Constructor Summary

Constructors

Constructor and Description
HFileCorruptionChecker(org.apache.hadoop.conf.Configuration conf, ExecutorService executor, boolean quarantine)

Method Summary

Modifier and Type	Method and Description
protected void	checkColFamDir(org.apache.hadoop.fs.Path cfDir) Check all files in a column family dir.
protected void	checkHFile(org.apache.hadoop.fs.Path p) Checks a path to see if it is a valid hfile.
protected void	checkMobColFamDir(org.apache.hadoop.fs.Path cfDir) Check all files in a mob column family dir.
protected void	checkMobFile(org.apache.hadoop.fs.Path p) Checks a path to see if it is a valid mob file.
private void	checkMobRegionDir(org.apache.hadoop.fs.Path regionDir) Checks all the mob files of a table.
protected void	checkRegionDir(org.apache.hadoop.fs.Path regionDir) Check all column families in a region dir.
(package private) void	checkTableDir(org.apache.hadoop.fs.Path tableDir) Check all the regiondirs in the specified tableDir
void	checkTables(Collection<org.apache.hadoop.fs.Path> tables) Check the specified table dirs for bad hfiles.
private HFileCorruptionChecker.MobRegionDirChecker	createMobRegionDirChecker(org.apache.hadoop.fs.Path tableDir) Creates an instance of MobRegionDirChecker.
(package private) org.apache.hadoop.fs.Path	createQuarantinePath(org.apache.hadoop.fs.Path hFile) Given a path, generates a new path to where we move a corrupted hfile (bad trailer, no trailer).
Collection<org.apache.hadoop.fs.Path>	getCorrupted()
Collection<org.apache.hadoop.fs.Path>	getCorruptedMobFiles()
Collection<org.apache.hadoop.fs.Path>	getFailureMobFiles()
Collection<org.apache.hadoop.fs.Path>	getFailures()
int	getHFilesChecked()
Collection<org.apache.hadoop.fs.Path>	getMissedMobFiles()
Collection<org.apache.hadoop.fs.Path>	getMissing()
int	getMobFilesChecked()
Collection<org.apache.hadoop.fs.Path>	getQuarantined()
Collection<org.apache.hadoop.fs.Path>	getQuarantinedMobFiles()
void	report(HbckErrorReporter out) Print a human readable summary of hfile quarantining operations.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

private static final org.slf4j.Logger LOG

conf

final org.apache.hadoop.conf.Configuration conf

fs

final org.apache.hadoop.fs.FileSystem fs

cacheConf

final CacheConfig cacheConf

executor

final ExecutorService executor

corrupted

final Set<org.apache.hadoop.fs.Path> corrupted

failures

final Set<org.apache.hadoop.fs.Path> failures

quarantined

final Set<org.apache.hadoop.fs.Path> quarantined

missing

final Set<org.apache.hadoop.fs.Path> missing

corruptedMobFiles

final Set<org.apache.hadoop.fs.Path> corruptedMobFiles

failureMobFiles

final Set<org.apache.hadoop.fs.Path> failureMobFiles

missedMobFiles

final Set<org.apache.hadoop.fs.Path> missedMobFiles

quarantinedMobFiles

final Set<org.apache.hadoop.fs.Path> quarantinedMobFiles

inQuarantineMode

final boolean inQuarantineMode

hfilesChecked

final AtomicInteger hfilesChecked

mobFilesChecked

final AtomicInteger mobFilesChecked

Constructor Detail

HFileCorruptionChecker

public HFileCorruptionChecker(org.apache.hadoop.conf.Configuration conf,
                              ExecutorService executor,
                              boolean quarantine)
                       throws IOException

Throws:

IOException

Method Detail

checkHFile

protected void checkHFile(org.apache.hadoop.fs.Path p)
                   throws IOException

Checks a path to see if it is a valid hfile.

Parameters:

p - full Path to an HFile

Throws:

IOException - This is a connectivity related exception

createQuarantinePath

org.apache.hadoop.fs.Path createQuarantinePath(org.apache.hadoop.fs.Path hFile)
                                        throws IOException

Given a path, generates a new path to where we move a corrupted hfile (bad trailer, no trailer).

Parameters:

hFile - Path to a corrupt hfile (assumes that it is HBASE_DIR/ table /region/cf/file)

Returns:

path to where corrupted files are stored. This should be HBASE_DIR/.corrupt/table/region/cf/file.

Throws:

IOException

checkColFamDir

protected void checkColFamDir(org.apache.hadoop.fs.Path cfDir)
                       throws IOException

Check all files in a column family dir.

Parameters:

cfDir - column family directory

Throws:

IOException

checkMobColFamDir

protected void checkMobColFamDir(org.apache.hadoop.fs.Path cfDir)
                          throws IOException

Check all files in a mob column family dir.

Parameters:

cfDir - mob column family directory

Throws:

IOException

checkMobFile

protected void checkMobFile(org.apache.hadoop.fs.Path p)
                     throws IOException

Checks a path to see if it is a valid mob file.

Parameters:

p - full Path to a mob file.

Throws:

IOException - This is a connectivity related exception

checkMobRegionDir

private void checkMobRegionDir(org.apache.hadoop.fs.Path regionDir)
                        throws IOException

Checks all the mob files of a table.

Parameters:

regionDir - The mob region directory

Throws:

IOException

checkRegionDir

protected void checkRegionDir(org.apache.hadoop.fs.Path regionDir)
                       throws IOException

Check all column families in a region dir.

Parameters:

regionDir - region directory

Throws:

IOException

checkTableDir

void checkTableDir(org.apache.hadoop.fs.Path tableDir)
            throws IOException

Check all the regiondirs in the specified tableDir

Parameters:

tableDir - path to a table

Throws:

IOException

createMobRegionDirChecker

private HFileCorruptionChecker.MobRegionDirChecker createMobRegionDirChecker(org.apache.hadoop.fs.Path tableDir)

Creates an instance of MobRegionDirChecker.

Parameters:

tableDir - The current table directory.

Returns:

An instance of MobRegionDirChecker.

checkTables

public void checkTables(Collection<org.apache.hadoop.fs.Path> tables)
                 throws IOException

Check the specified table dirs for bad hfiles.

Throws:

IOException

getFailures

public Collection<org.apache.hadoop.fs.Path> getFailures()

Returns:

the set of check failure file paths after checkTables is called.

getCorrupted

public Collection<org.apache.hadoop.fs.Path> getCorrupted()

Returns:

the set of corrupted file paths after checkTables is called.

getHFilesChecked

public int getHFilesChecked()

Returns:

number of hfiles checked in the last HfileCorruptionChecker run

getQuarantined

public Collection<org.apache.hadoop.fs.Path> getQuarantined()

Returns:

the set of successfully quarantined paths after checkTables is called.

getMissing

public Collection<org.apache.hadoop.fs.Path> getMissing()

Returns:

the set of paths that were missing. Likely due to deletion/moves from compaction or flushes.

getFailureMobFiles

public Collection<org.apache.hadoop.fs.Path> getFailureMobFiles()

Returns:

the set of check failure mob file paths after checkTables is called.

getCorruptedMobFiles

public Collection<org.apache.hadoop.fs.Path> getCorruptedMobFiles()

Returns:

the set of corrupted mob file paths after checkTables is called.

getMobFilesChecked

public int getMobFilesChecked()

Returns:

number of mob files checked in the last HfileCorruptionChecker run

getQuarantinedMobFiles

public Collection<org.apache.hadoop.fs.Path> getQuarantinedMobFiles()

Returns:

the set of successfully quarantined paths after checkTables is called.

getMissedMobFiles

public Collection<org.apache.hadoop.fs.Path> getMissedMobFiles()

Returns:

the set of paths that were missing. Likely due to table deletion or deletion/moves from compaction.

report

public void report(HbckErrorReporter out)

Print a human readable summary of hfile quarantining operations.

Parameters:

out -