org.apache.hadoop.hbase.master.cleaner

Class CleanerChore<T extends FileCleanerDelegate>

java.lang.Object

org.apache.hadoop.hbase.ScheduledChore

org.apache.hadoop.hbase.master.cleaner.CleanerChore<T>

Type Parameters:

T - Cleaner delegate class that is dynamically loaded from configuration

All Implemented Interfaces:

Runnable

Direct Known Subclasses:

HFileCleaner, LogCleaner

@InterfaceAudience.Private
public abstract class CleanerChore<T extends FileCleanerDelegate>
extends ScheduledChore

Abstract Cleaner that uses a chain of delegates to clean a directory of files

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
private static interface	CleanerChore.Action<T>

Field Summary

Fields

Modifier and Type	Field and Description
private static int	AVAIL_PROCESSORS
static String	CHORE_POOL_SIZE Configures the threadpool used for scanning the archive directory for the HFileCleaner If it is an integer and >= 1, it would be the size; if 0.0 < size <= 1.0, size would be available processors * size.
protected List<T>	cleanersChain
private org.apache.hadoop.conf.Configuration	conf
(package private) static String	DEFAULT_CHORE_POOL_SIZE
(package private) static String	DEFAULT_LOG_CLEANER_CHORE_POOL_SIZE
private AtomicBoolean	enabled
protected List<String>	excludeDirs
protected org.apache.hadoop.fs.FileSystem	fs
private static org.slf4j.Logger	LOG
static String	LOG_CLEANER_CHORE_SIZE Configures the threadpool used for scanning the Old logs directory for the LogCleaner Follows the same configuration mechanism as CHORE_POOL_SIZE, but has a default of 1 thread.
private org.apache.hadoop.fs.Path	oldFileDir
protected Map<String,Object>	params
private DirScanPool	pool

Constructor Summary

Constructors

Constructor and Description
CleanerChore(String name, int sleepPeriod, Stoppable s, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path oldFileDir, String confKey, DirScanPool pool)
CleanerChore(String name, int sleepPeriod, Stoppable s, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path oldFileDir, String confKey, DirScanPool pool, Map<String,Object> params, List<org.apache.hadoop.fs.Path> excludePaths)

Method Summary

Modifier and Type	Method and Description
(package private) static int	calculatePoolSize(String poolSize) Calculate size for cleaner pool.
private boolean	checkAndDeleteFiles(List<org.apache.hadoop.fs.FileStatus> files) Run the given files through each of the cleaners to see if it should be deleted, deleting it if necessary.
protected void	chore() The task to execute on each scheduled execution of the Chore
void	cleanup() Override to run cleanup tasks when the Chore encounters an error and must stop running
private boolean	deleteAction(CleanerChore.Action<Boolean> deletion, String type, org.apache.hadoop.fs.Path dir) Perform a delete on a specified type.
protected int	deleteFiles(Iterable<org.apache.hadoop.fs.FileStatus> filesToDelete) Delete the given files
(package private) int	getChorePoolSize()
boolean	getEnabled()
private void	initCleanerChain(String confKey) Instantiate and initialize all the file cleaners set in the configuration
private boolean	isEmptyDirDeletable(org.apache.hadoop.fs.Path dir) Check if a empty directory with no subdirs or subfiles can be deleted
private T	newFileCleaner(String className, org.apache.hadoop.conf.Configuration conf) A utility method to create new instances of LogCleanerDelegate based on the class name of the LogCleanerDelegate.
private void	preRunCleaner()
boolean	runCleaner()
boolean	setEnabled(boolean enabled) n
private boolean	shouldExclude(org.apache.hadoop.fs.FileStatus f) Check if a path should not perform clear
private void	sortByConsumedSpace(List<org.apache.hadoop.fs.FileStatus> dirs) Sort the given list in (descending) order of the space each element takes
private void	traverseAndDelete(org.apache.hadoop.fs.Path dir, boolean root, CompletableFuture<Boolean> result) Attempts to clean up a directory(its subdirectories, and files) in a ThreadPoolExecutor concurrently.
protected abstract boolean	validate(org.apache.hadoop.fs.Path file) Validate the file to see if it even belongs in the directory.

Methods inherited from class org.apache.hadoop.hbase.ScheduledChore

cancel, cancel, choreForTesting, getInitialDelay, getName, getPeriod, getStopper, getTimeUnit, initialChore, isInitialChoreComplete, isScheduled, run, shutdown, shutdown, toString, triggerNow

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

LOG

private static final org.slf4j.Logger LOG

AVAIL_PROCESSORS

private static final int AVAIL_PROCESSORS

CHORE_POOL_SIZE

public static final String CHORE_POOL_SIZE

Configures the threadpool used for scanning the archive directory for the HFileCleaner If it is an integer and >= 1, it would be the size; if 0.0 < size <= 1.0, size would be available processors * size. Pay attention that 1.0 is different from 1, former indicates it will use 100% of cores, while latter will use only 1 thread for chore to scan dir.

See Also:

Constant Field Values

DEFAULT_CHORE_POOL_SIZE

static final String DEFAULT_CHORE_POOL_SIZE

See Also:

Constant Field Values

LOG_CLEANER_CHORE_SIZE

public static final String LOG_CLEANER_CHORE_SIZE

Configures the threadpool used for scanning the Old logs directory for the LogCleaner Follows the same configuration mechanism as CHORE_POOL_SIZE, but has a default of 1 thread.

See Also:

Constant Field Values

DEFAULT_LOG_CLEANER_CHORE_POOL_SIZE

static final String DEFAULT_LOG_CLEANER_CHORE_POOL_SIZE

See Also:

Constant Field Values

pool

private final DirScanPool pool

fs

protected final org.apache.hadoop.fs.FileSystem fs

oldFileDir

private final org.apache.hadoop.fs.Path oldFileDir

conf

private final org.apache.hadoop.conf.Configuration conf

params

protected final Map<String,Object> params

enabled

private final AtomicBoolean enabled

cleanersChain

protected List<T extends FileCleanerDelegate> cleanersChain

excludeDirs

protected List<String> excludeDirs

Constructor Detail

CleanerChore

public CleanerChore(String name,
                    int sleepPeriod,
                    Stoppable s,
                    org.apache.hadoop.conf.Configuration conf,
                    org.apache.hadoop.fs.FileSystem fs,
                    org.apache.hadoop.fs.Path oldFileDir,
                    String confKey,
                    DirScanPool pool)

CleanerChore

public CleanerChore(String name,
                    int sleepPeriod,
                    Stoppable s,
                    org.apache.hadoop.conf.Configuration conf,
                    org.apache.hadoop.fs.FileSystem fs,
                    org.apache.hadoop.fs.Path oldFileDir,
                    String confKey,
                    DirScanPool pool,
                    Map<String,Object> params,
                    List<org.apache.hadoop.fs.Path> excludePaths)

Parameters:

name - name of the chore being run

sleepPeriod - the period of time to sleep between each run

s - the stopper

conf - configuration to use

fs - handle to the FS

oldFileDir - the path to the archived files

confKey - configuration key for the classes to instantiate

pool - the thread pool used to scan directories

params - members could be used in cleaner

Method Detail

calculatePoolSize

static int calculatePoolSize(String poolSize)

Calculate size for cleaner pool.

Parameters:

poolSize - size from configuration

Returns:

size of pool after calculation

validate

protected abstract boolean validate(org.apache.hadoop.fs.Path file)

Validate the file to see if it even belongs in the directory. If it is valid, then the file will go through the cleaner delegates, but otherwise the file is just deleted.

Parameters:

file - full Path of the file to be checked

Returns:

true if the file is valid, false otherwise

initCleanerChain

private void initCleanerChain(String confKey)

Instantiate and initialize all the file cleaners set in the configuration

Parameters:

confKey - key to get the file cleaner classes from the configuration

newFileCleaner

private T newFileCleaner(String className,
                         org.apache.hadoop.conf.Configuration conf)

A utility method to create new instances of LogCleanerDelegate based on the class name of the LogCleanerDelegate.

Parameters:

className - fully qualified class name of the LogCleanerDelegate

conf - used configuration

Returns:

the new instance

chore

protected void chore()

Description copied from class: ScheduledChore

The task to execute on each scheduled execution of the Chore

Specified by:

chore in class ScheduledChore

preRunCleaner

private void preRunCleaner()

runCleaner

public boolean runCleaner()

sortByConsumedSpace

private void sortByConsumedSpace(List<org.apache.hadoop.fs.FileStatus> dirs)

Sort the given list in (descending) order of the space each element takes

Parameters:

dirs - the list to sort, element in it should be directory (not file)

checkAndDeleteFiles

private boolean checkAndDeleteFiles(List<org.apache.hadoop.fs.FileStatus> files)

Run the given files through each of the cleaners to see if it should be deleted, deleting it if necessary.

Parameters:

files - List of FileStatus for the files to check (and possibly delete)

Returns:

true iff successfully deleted all files

isEmptyDirDeletable

private boolean isEmptyDirDeletable(org.apache.hadoop.fs.Path dir)

Check if a empty directory with no subdirs or subfiles can be deleted

Parameters:

dir - Path of the directory

Returns:

True if the directory can be deleted, otherwise false

deleteFiles

protected int deleteFiles(Iterable<org.apache.hadoop.fs.FileStatus> filesToDelete)

Delete the given files

Parameters:

filesToDelete - files to delete

Returns:

number of deleted files

cleanup

public void cleanup()

Description copied from class: ScheduledChore

Override to run cleanup tasks when the Chore encounters an error and must stop running

Overrides:

cleanup in class ScheduledChore

getChorePoolSize

int getChorePoolSize()

setEnabled

public boolean setEnabled(boolean enabled)

n

getEnabled

public boolean getEnabled()

traverseAndDelete

private void traverseAndDelete(org.apache.hadoop.fs.Path dir,
                               boolean root,
                               CompletableFuture<Boolean> result)

Attempts to clean up a directory(its subdirectories, and files) in a ThreadPoolExecutor concurrently. We can get the final result by calling result.get().

shouldExclude

private boolean shouldExclude(org.apache.hadoop.fs.FileStatus f)

Check if a path should not perform clear

deleteAction

private boolean deleteAction(CleanerChore.Action<Boolean> deletion,
                             String type,
                             org.apache.hadoop.fs.Path dir)

Perform a delete on a specified type.

Parameters:

deletion - a delete

type - possible values are 'files', 'subdirs', 'dirs'

Returns:

true if it deleted successfully, false otherwise