org.apache.hadoop.hbase.master

Class SplitLogManager

java.lang.Object

org.apache.hadoop.hbase.master.SplitLogManager

@InterfaceAudience.Private
public class SplitLogManager
extends Object

Distributes the task of log splitting to the available region servers. Coordination happens via coordination engine. For every log file that has to be split a task is created. SplitLogWorkers race to grab a task.

SplitLogManager monitors the tasks that it creates using the timeoutMonitor thread. If a task's progress is slow then SplitLogManagerCoordination.checkTasks() will take away the task from the owner SplitLogWorker and the task will be up for grabs again. When the task is done then it is deleted by SplitLogManager.

Clients call splitLogDistributed(Path) to split a region server's log files. The caller thread waits in this method until all the log files have been split.

All the coordination calls made by this class are asynchronous. This is mainly to help reduce response time seen by the callers.

There is race in this design between the SplitLogManager and the SplitLogWorker. SplitLogManager might re-queue a task that has in reality already been completed by a SplitLogWorker. We rely on the idempotency of the log splitting task for correctness.

It is also assumed that every log splitting task is unique and once completed (either with success or with error) it will be not be submitted again. If a task is resubmitted then there is a risk that old "delete task" can delete the re-submission.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	SplitLogManager.ResubmitDirective
static class	SplitLogManager.Task in memory state of an active task.
static class	SplitLogManager.TaskBatch Keeps track of the batch of tasks submitted together by a caller in splitLogDistributed().
static class	SplitLogManager.TerminationStatus

Field Summary

Fields

Modifier and Type	Field and Description
static int	DEFAULT_UNASSIGNED_TIMEOUT
protected ReentrantLock	recoveringRegionLock In distributedLogReplay mode, we need touch both splitlog and recovering-regions znodes in one operation.

Constructor Summary

Constructors

Constructor and Description
SplitLogManager(Server server, org.apache.hadoop.conf.Configuration conf, Stoppable stopper, MasterServices master, ServerName serverName) Its OK to construct this object even when region-servers are not online.

Method Summary

Methods

Modifier and Type	Method and Description
static org.apache.hadoop.fs.FileStatus[]	getFileList(org.apache.hadoop.conf.Configuration conf, List<org.apache.hadoop.fs.Path> logDirs, org.apache.hadoop.fs.PathFilter filter) Get a list of paths that need to be split given a set of server-specific directories and optionally a filter.
ZooKeeperProtos.SplitLogTask.RecoveryMode	getRecoveryMode()
boolean	isLogReplaying()
boolean	isLogSplitting()
void	markRegionsRecovering(ServerName server, Set<HRegionInfo> userRegions)
void	setRecoveryMode(boolean isForInitialization) This function is to set recovery mode from outstanding split log tasks from before or current configuration setting
long	splitLogDistributed(List<org.apache.hadoop.fs.Path> logDirs) The caller will block until all the log files of the given region server have been processed - successfully split or an error is encountered - by an available worker region server.
long	splitLogDistributed(org.apache.hadoop.fs.Path logDir)
long	splitLogDistributed(Set<ServerName> serverNames, List<org.apache.hadoop.fs.Path> logDirs, org.apache.hadoop.fs.PathFilter filter) The caller will block until all the hbase:meta log files of the given region server have been processed - successfully split or an error is encountered - by an available worker region server.
void	stop()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_UNASSIGNED_TIMEOUT

public static final int DEFAULT_UNASSIGNED_TIMEOUT

See Also:

Constant Field Values

recoveringRegionLock

protected final ReentrantLock recoveringRegionLock

In distributedLogReplay mode, we need touch both splitlog and recovering-regions znodes in one operation. So the lock is used to guard such cases.

Constructor Detail

SplitLogManager

public SplitLogManager(Server server,
               org.apache.hadoop.conf.Configuration conf,
               Stoppable stopper,
               MasterServices master,
               ServerName serverName)
                throws IOException

Its OK to construct this object even when region-servers are not online. It does lookup the orphan tasks in coordination engine but it doesn't block waiting for them to be done.

Parameters:

server - the server instance

conf - the HBase configuration

stopper - the stoppable in case anything is wrong

master - the master services

serverName - the master server name

Throws:

IOException

Method Detail

getFileList

public static org.apache.hadoop.fs.FileStatus[] getFileList(org.apache.hadoop.conf.Configuration conf,
                                            List<org.apache.hadoop.fs.Path> logDirs,
                                            org.apache.hadoop.fs.PathFilter filter)
                                                     throws IOException

Get a list of paths that need to be split given a set of server-specific directories and optionally a filter. See DefaultWALProvider.getServerNameFromWALDirectoryName(org.apache.hadoop.conf.Configuration, java.lang.String) for more info on directory layout. Should be package-private, but is needed by WALSplitter.split(Path, Path, Path, FileSystem, Configuration, WALFactory) for tests.

Throws:

IOException

splitLogDistributed

public long splitLogDistributed(org.apache.hadoop.fs.Path logDir)
                         throws IOException

Parameters:

logDir - one region sever wal dir path in .logs

Returns:

cumulative size of the logfiles split

Throws:

IOException - if there was an error while splitting any log file

IOException

splitLogDistributed

public long splitLogDistributed(List<org.apache.hadoop.fs.Path> logDirs)
                         throws IOException

The caller will block until all the log files of the given region server have been processed - successfully split or an error is encountered - by an available worker region server. This method must only be called after the region servers have been brought online.

Parameters:

logDirs - List of log dirs to split

Returns:

cumulative size of the logfiles split

Throws:

IOException - If there was an error while splitting any log file

splitLogDistributed

public long splitLogDistributed(Set<ServerName> serverNames,
                       List<org.apache.hadoop.fs.Path> logDirs,
                       org.apache.hadoop.fs.PathFilter filter)
                         throws IOException

The caller will block until all the hbase:meta log files of the given region server have been processed - successfully split or an error is encountered - by an available worker region server. This method must only be called after the region servers have been brought online.

Parameters:

logDirs - List of log dirs to split

filter - the Path filter to select specific files for considering

Returns:

cumulative size of the logfiles split

Throws:

IOException - If there was an error while splitting any log file

stop

public void stop()

setRecoveryMode

public void setRecoveryMode(boolean isForInitialization)
                     throws IOException

This function is to set recovery mode from outstanding split log tasks from before or current configuration setting

Parameters:

isForInitialization -

Throws:

IOException - throws if it's impossible to set recovery mode

markRegionsRecovering

public void markRegionsRecovering(ServerName server,
                         Set<HRegionInfo> userRegions)
                           throws InterruptedIOException,
                                  IOException

Throws:

InterruptedIOException

IOException

isLogReplaying

public boolean isLogReplaying()

Returns:

whether log is replaying

isLogSplitting

public boolean isLogSplitting()

Returns:

whether log is splitting

getRecoveryMode

public ZooKeeperProtos.SplitLogTask.RecoveryMode getRecoveryMode()

Returns:

the current log recovery mode