org.apache.hadoop.hbase.master.cleaner

Class HFileCleaner

java.lang.Object

org.apache.hadoop.hbase.ScheduledChore

org.apache.hadoop.hbase.master.cleaner.CleanerChore<BaseHFileCleanerDelegate>

org.apache.hadoop.hbase.master.cleaner.HFileCleaner

All Implemented Interfaces:

Runnable, ConfigurationObserver

@InterfaceAudience.Private
public class HFileCleaner
extends CleanerChore<BaseHFileCleanerDelegate>
implements ConfigurationObserver

This Chore, every time it runs, will clear the HFiles in the hfile archive folder that are deletable for each HFile cleaner in the chain.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
private static class	HFileCleaner.HFileDeleteTask

Field Summary

Fields

Modifier and Type	Field and Description
private long	cleanerThreadCheckIntervalMsec
private long	cleanerThreadTimeoutMsec
private static Comparator<HFileCleaner.HFileDeleteTask>	COMPARATOR
static String	CUSTOM_POOL_SIZE
(package private) static long	DEFAULT_HFILE_DELETE_THREAD_CHECK_INTERVAL_MSEC
(package private) static long	DEFAULT_HFILE_DELETE_THREAD_TIMEOUT_MSEC
static int	DEFAULT_HFILE_DELETE_THROTTLE_THRESHOLD
static int	DEFAULT_LARGE_HFILE_DELETE_THREAD_NUMBER
static int	DEFAULT_LARGE_HFILE_QUEUE_INIT_SIZE
static int	DEFAULT_SMALL_HFILE_DELETE_THREAD_NUMBER
static int	DEFAULT_SMALL_HFILE_QUEUE_INIT_SIZE
private AtomicLong	deletedLargeFiles
private AtomicLong	deletedSmallFiles
static String	HFILE_CLEANER_CUSTOM_PATHS The custom paths for hfile cleaner, subdirectories of archive, e.g.
static String	HFILE_CLEANER_CUSTOM_PATHS_PLUGINS Configure hfile cleaner classes for the custom paths
static String	HFILE_DELETE_THREAD_CHECK_INTERVAL_MSEC
static String	HFILE_DELETE_THREAD_TIMEOUT_MSEC
static String	HFILE_DELETE_THROTTLE_THRESHOLD
static String	LARGE_HFILE_DELETE_THREAD_NUMBER
static String	LARGE_HFILE_QUEUE_INIT_SIZE
private int	largeFileDeleteThreadNumber
(package private) StealJobQueue<HFileCleaner.HFileDeleteTask>	largeFileQueue
private int	largeQueueInitSize
private static org.slf4j.Logger	LOG
static String	MASTER_HFILE_CLEANER_PLUGINS
private boolean	running
static String	SMALL_HFILE_DELETE_THREAD_NUMBER
static String	SMALL_HFILE_QUEUE_INIT_SIZE
private int	smallFileDeleteThreadNumber
(package private) BlockingQueue<HFileCleaner.HFileDeleteTask>	smallFileQueue
private int	smallQueueInitSize
private List<Thread>	threads
private int	throttlePoint

Fields inherited from class org.apache.hadoop.hbase.master.cleaner.CleanerChore

CHORE_POOL_SIZE, cleanersChain, DEFAULT_CHORE_POOL_SIZE, DEFAULT_LOG_CLEANER_CHORE_POOL_SIZE, excludeDirs, fs, LOG_CLEANER_CHORE_SIZE, params

Constructor Summary

Constructors

Constructor and Description
HFileCleaner(int period, Stoppable stopper, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path directory, DirScanPool pool)
HFileCleaner(int period, Stoppable stopper, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path directory, DirScanPool pool, Map<String,Object> params)
HFileCleaner(int period, Stoppable stopper, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path directory, DirScanPool pool, Map<String,Object> params, List<org.apache.hadoop.fs.Path> excludePaths)
HFileCleaner(String name, int period, Stoppable stopper, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path directory, String confKey, DirScanPool pool, Map<String,Object> params, List<org.apache.hadoop.fs.Path> excludePaths) For creating customized HFileCleaner.

Method Summary

Modifier and Type	Method and Description
void	cancel(boolean mayInterruptIfRunning)
private boolean	checkAndUpdateConfigurations(org.apache.hadoop.conf.Configuration conf) Check new configuration and update settings if value changed
void	cleanup() Override to run cleanup tasks when the Chore encounters an error and must stop running
protected void	consumerLoop(BlockingQueue<HFileCleaner.HFileDeleteTask> queue)
private void	countDeletedFiles(boolean isLargeFile, boolean fromLargeQueue)
private HFileCleaner.HFileDeleteTask	deleteFile(org.apache.hadoop.fs.FileStatus file) Construct an HFileCleaner.HFileDeleteTask for each file to delete and add into the correct queue
int	deleteFiles(Iterable<org.apache.hadoop.fs.FileStatus> filesToDelete) Delete the given files
private boolean	dispatch(HFileCleaner.HFileDeleteTask task)
(package private) long	getCleanerThreadCheckIntervalMsec()
List<Thread>	getCleanerThreads()
(package private) long	getCleanerThreadTimeoutMsec()
List<BaseHFileCleanerDelegate>	getDelegatesForTesting() Exposed for TESTING!
long	getLargeQueueInitSize()
long	getNumOfDeletedLargeFiles()
long	getNumOfDeletedSmallFiles()
long	getSmallQueueInitSize()
long	getThrottlePoint()
void	onConfigurationChange(org.apache.hadoop.conf.Configuration conf) This method would be called by the ConfigurationManager object when the Configuration object is reloaded from disk.
private void	startHFileDeleteThreads() Start threads for hfile deletion
private void	stopHFileDeleteThreads() Stop threads for hfile deletion
protected boolean	validate(org.apache.hadoop.fs.Path file) Validate the file to see if it even belongs in the directory.

Methods inherited from class org.apache.hadoop.hbase.master.cleaner.CleanerChore

calculatePoolSize, chore, getChorePoolSize, getEnabled, runCleaner, setEnabled

Methods inherited from class org.apache.hadoop.hbase.ScheduledChore

cancel, choreForTesting, getInitialDelay, getName, getPeriod, getStopper, getTimeUnit, initialChore, isInitialChoreComplete, isScheduled, run, shutdown, shutdown, toString, triggerNow

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

MASTER_HFILE_CLEANER_PLUGINS

public static final String MASTER_HFILE_CLEANER_PLUGINS

See Also:

Constant Field Values

HFILE_DELETE_THROTTLE_THRESHOLD

public static final String HFILE_DELETE_THROTTLE_THRESHOLD

See Also:

Constant Field Values

DEFAULT_HFILE_DELETE_THROTTLE_THRESHOLD

public static final int DEFAULT_HFILE_DELETE_THROTTLE_THRESHOLD

See Also:

Constant Field Values

LARGE_HFILE_QUEUE_INIT_SIZE

public static final String LARGE_HFILE_QUEUE_INIT_SIZE

See Also:

Constant Field Values

DEFAULT_LARGE_HFILE_QUEUE_INIT_SIZE

public static final int DEFAULT_LARGE_HFILE_QUEUE_INIT_SIZE

See Also:

Constant Field Values

SMALL_HFILE_QUEUE_INIT_SIZE

public static final String SMALL_HFILE_QUEUE_INIT_SIZE

See Also:

Constant Field Values

DEFAULT_SMALL_HFILE_QUEUE_INIT_SIZE

public static final int DEFAULT_SMALL_HFILE_QUEUE_INIT_SIZE

See Also:

Constant Field Values

LARGE_HFILE_DELETE_THREAD_NUMBER

public static final String LARGE_HFILE_DELETE_THREAD_NUMBER

See Also:

Constant Field Values

DEFAULT_LARGE_HFILE_DELETE_THREAD_NUMBER

public static final int DEFAULT_LARGE_HFILE_DELETE_THREAD_NUMBER

See Also:

Constant Field Values

SMALL_HFILE_DELETE_THREAD_NUMBER

public static final String SMALL_HFILE_DELETE_THREAD_NUMBER

See Also:

Constant Field Values

DEFAULT_SMALL_HFILE_DELETE_THREAD_NUMBER

public static final int DEFAULT_SMALL_HFILE_DELETE_THREAD_NUMBER

See Also:

Constant Field Values

HFILE_DELETE_THREAD_TIMEOUT_MSEC

public static final String HFILE_DELETE_THREAD_TIMEOUT_MSEC

See Also:

Constant Field Values

DEFAULT_HFILE_DELETE_THREAD_TIMEOUT_MSEC

static final long DEFAULT_HFILE_DELETE_THREAD_TIMEOUT_MSEC

See Also:

Constant Field Values

HFILE_DELETE_THREAD_CHECK_INTERVAL_MSEC

public static final String HFILE_DELETE_THREAD_CHECK_INTERVAL_MSEC

See Also:

Constant Field Values

DEFAULT_HFILE_DELETE_THREAD_CHECK_INTERVAL_MSEC

static final long DEFAULT_HFILE_DELETE_THREAD_CHECK_INTERVAL_MSEC

See Also:

Constant Field Values

HFILE_CLEANER_CUSTOM_PATHS

public static final String HFILE_CLEANER_CUSTOM_PATHS

The custom paths for hfile cleaner, subdirectories of archive, e.g. data/default/testTable1,data/default/testTable2

See Also:

Constant Field Values

HFILE_CLEANER_CUSTOM_PATHS_PLUGINS

public static final String HFILE_CLEANER_CUSTOM_PATHS_PLUGINS

Configure hfile cleaner classes for the custom paths

See Also:

Constant Field Values

CUSTOM_POOL_SIZE

public static final String CUSTOM_POOL_SIZE

See Also:

Constant Field Values

LOG

private static final org.slf4j.Logger LOG

largeFileQueue

StealJobQueue<HFileCleaner.HFileDeleteTask> largeFileQueue

smallFileQueue

BlockingQueue<HFileCleaner.HFileDeleteTask> smallFileQueue

throttlePoint

private int throttlePoint

largeQueueInitSize

private int largeQueueInitSize

smallQueueInitSize

private int smallQueueInitSize

largeFileDeleteThreadNumber

private int largeFileDeleteThreadNumber

smallFileDeleteThreadNumber

private int smallFileDeleteThreadNumber

cleanerThreadTimeoutMsec

private long cleanerThreadTimeoutMsec

cleanerThreadCheckIntervalMsec

private long cleanerThreadCheckIntervalMsec

threads

private List<Thread> threads

running

private volatile boolean running

deletedLargeFiles

private AtomicLong deletedLargeFiles

deletedSmallFiles

private AtomicLong deletedSmallFiles

COMPARATOR

private static final Comparator<HFileCleaner.HFileDeleteTask> COMPARATOR

Constructor Detail

HFileCleaner

public HFileCleaner(int period,
                    Stoppable stopper,
                    org.apache.hadoop.conf.Configuration conf,
                    org.apache.hadoop.fs.FileSystem fs,
                    org.apache.hadoop.fs.Path directory,
                    DirScanPool pool)

HFileCleaner

public HFileCleaner(int period,
                    Stoppable stopper,
                    org.apache.hadoop.conf.Configuration conf,
                    org.apache.hadoop.fs.FileSystem fs,
                    org.apache.hadoop.fs.Path directory,
                    DirScanPool pool,
                    Map<String,Object> params)

Parameters:

period - the period of time to sleep between each run

stopper - the stopper

conf - configuration to use

fs - handle to the FS

directory - directory to be cleaned

pool - the thread pool used to scan directories

params - params could be used in subclass of BaseHFileCleanerDelegate

HFileCleaner

public HFileCleaner(int period,
                    Stoppable stopper,
                    org.apache.hadoop.conf.Configuration conf,
                    org.apache.hadoop.fs.FileSystem fs,
                    org.apache.hadoop.fs.Path directory,
                    DirScanPool pool,
                    Map<String,Object> params,
                    List<org.apache.hadoop.fs.Path> excludePaths)

HFileCleaner

public HFileCleaner(String name,
                    int period,
                    Stoppable stopper,
                    org.apache.hadoop.conf.Configuration conf,
                    org.apache.hadoop.fs.FileSystem fs,
                    org.apache.hadoop.fs.Path directory,
                    String confKey,
                    DirScanPool pool,
                    Map<String,Object> params,
                    List<org.apache.hadoop.fs.Path> excludePaths)

For creating customized HFileCleaner.

Parameters:

name - name of the chore being run

period - the period of time to sleep between each run

stopper - the stopper

conf - configuration to use

fs - handle to the FS

directory - directory to be cleaned

confKey - configuration key for the classes to instantiate

pool - the thread pool used to scan directories

params - params could be used in subclass of BaseHFileCleanerDelegate

Method Detail

validate

protected boolean validate(org.apache.hadoop.fs.Path file)

Description copied from class: CleanerChore

Validate the file to see if it even belongs in the directory. If it is valid, then the file will go through the cleaner delegates, but otherwise the file is just deleted.

Specified by:

validate in class CleanerChore<BaseHFileCleanerDelegate>

Parameters:

file - full Path of the file to be checked

Returns:

true if the file is valid, false otherwise

getDelegatesForTesting

public List<BaseHFileCleanerDelegate> getDelegatesForTesting()

Exposed for TESTING!

deleteFiles

public int deleteFiles(Iterable<org.apache.hadoop.fs.FileStatus> filesToDelete)

Description copied from class: CleanerChore

Delete the given files

Overrides:

deleteFiles in class CleanerChore<BaseHFileCleanerDelegate>

Parameters:

filesToDelete - files to delete

Returns:

number of deleted files

deleteFile

private HFileCleaner.HFileDeleteTask deleteFile(org.apache.hadoop.fs.FileStatus file)

Construct an HFileCleaner.HFileDeleteTask for each file to delete and add into the correct queue

Parameters:

file - the file to delete

Returns:

HFileDeleteTask to track progress

dispatch

private boolean dispatch(HFileCleaner.HFileDeleteTask task)

cleanup

public void cleanup()

Description copied from class: ScheduledChore

Override to run cleanup tasks when the Chore encounters an error and must stop running

Overrides:

cleanup in class CleanerChore<BaseHFileCleanerDelegate>

startHFileDeleteThreads

private void startHFileDeleteThreads()

Start threads for hfile deletion

consumerLoop

protected void consumerLoop(BlockingQueue<HFileCleaner.HFileDeleteTask> queue)

countDeletedFiles

private void countDeletedFiles(boolean isLargeFile,
                               boolean fromLargeQueue)

stopHFileDeleteThreads

private void stopHFileDeleteThreads()

Stop threads for hfile deletion

getCleanerThreads

public List<Thread> getCleanerThreads()

getNumOfDeletedLargeFiles

public long getNumOfDeletedLargeFiles()

getNumOfDeletedSmallFiles

public long getNumOfDeletedSmallFiles()

getLargeQueueInitSize

public long getLargeQueueInitSize()

getSmallQueueInitSize

public long getSmallQueueInitSize()

getThrottlePoint

public long getThrottlePoint()

getCleanerThreadTimeoutMsec

long getCleanerThreadTimeoutMsec()

getCleanerThreadCheckIntervalMsec

long getCleanerThreadCheckIntervalMsec()

onConfigurationChange

public void onConfigurationChange(org.apache.hadoop.conf.Configuration conf)

Description copied from interface: ConfigurationObserver

This method would be called by the ConfigurationManager object when the Configuration object is reloaded from disk.

Specified by:

onConfigurationChange in interface ConfigurationObserver

checkAndUpdateConfigurations

private boolean checkAndUpdateConfigurations(org.apache.hadoop.conf.Configuration conf)

Check new configuration and update settings if value changed

Parameters:

conf - The new configuration

Returns:

true if any configuration for HFileCleaner changes, false if no change

cancel

public void cancel(boolean mayInterruptIfRunning)

Overrides:

cancel in class ScheduledChore