SplitLogManager (Apache HBase 1.2.12 API)

java.lang.Object
- org.apache.hadoop.hbase.master.SplitLogManager

```
@InterfaceAudience.Private
public class SplitLogManager
extends Object
```
Distributes the task of log splitting to the available region servers. Coordination happens via coordination engine. For every log file that has to be split a task is created. SplitLogWorkers race to grab a task.
SplitLogManager monitors the tasks that it creates using the timeoutMonitor thread. If a task's progress is slow then SplitLogManagerCoordination.checkTasks() will take away the task from the owner SplitLogWorker and the task will be up for grabs again. When the task is done then it is deleted by SplitLogManager.
Clients call splitLogDistributed(Path) to split a region server's log files. The caller thread waits in this method until all the log files have been split.
All the coordination calls made by this class are asynchronous. This is mainly to help reduce response time seen by the callers.
There is race in this design between the SplitLogManager and the SplitLogWorker. SplitLogManager might re-queue a task that has in reality already been completed by a SplitLogWorker. We rely on the idempotency of the log splitting task for correctness.
It is also assumed that every log splitting task is unique and once completed (either with success or with error) it will be not be submitted again. If a task is resubmitted then there is a risk that old "delete task" can delete the re-submission.

Nested Class Summary

Nested Classes
Modifier and Type	Class and Description
`static class`	`SplitLogManager.ResubmitDirective`
`static class`	`SplitLogManager.Task` in memory state of an active task.
`static class`	`SplitLogManager.TaskBatch` Keeps track of the batch of tasks submitted together by a caller in splitLogDistributed().
`static class`	`SplitLogManager.TerminationStatus`
`private class`	`SplitLogManager.TimeoutMonitor` Periodically checks all active tasks and resubmits the ones that have timed out

Field Summary

Fields
Modifier and Type	Field and Description
`private long`	`checkRecoveringTimeThreshold`
`private ChoreService`	`choreService`
`private org.apache.hadoop.conf.Configuration`	`conf`
`private Set<ServerName>`	`deadWorkers`
`private Object`	`deadWorkersLock`
`static int`	`DEFAULT_UNASSIGNED_TIMEOUT`
`private List<Pair<Set<ServerName>,Boolean>>`	`failedRecoveringRegionDeletions`
`private long`	`lastTaskCreateTime`
`private static org.apache.commons.logging.Log`	`LOG`
`protected ReentrantLock`	`recoveringRegionLock` In distributedLogReplay mode, we need touch both splitlog and recovering-regions znodes in one operation.
`private Server`	`server`
`private Stoppable`	`stopper`
`private ConcurrentMap<String,SplitLogManager.Task>`	`tasks`
`private SplitLogManager.TimeoutMonitor`	`timeoutMonitor`
`private long`	`unassignedTimeout`

Constructor Summary

Constructors
Constructor and Description
`SplitLogManager(Server server, org.apache.hadoop.conf.Configuration conf, Stoppable stopper, MasterServices master, ServerName serverName)` Its OK to construct this object even when region-servers are not online.

Method Summary

Methods
Modifier and Type	Method and Description
`private int`	`activeTasks(SplitLogManager.TaskBatch batch)`
`private SplitLogManager.Task`	`createTaskIfAbsent(String path, SplitLogManager.TaskBatch batch)`
`(package private) boolean`	`enqueueSplitTask(String taskname, SplitLogManager.TaskBatch batch)` Add a task entry to coordination if it is not already there.
`(package private) SplitLogManager.Task`	`findOrCreateOrphanTask(String path)`
`static org.apache.hadoop.fs.FileStatus[]`	`getFileList(org.apache.hadoop.conf.Configuration conf, List<org.apache.hadoop.fs.Path> logDirs, org.apache.hadoop.fs.PathFilter filter)` Get a list of paths that need to be split given a set of server-specific directories and optionally a filter.
`private org.apache.hadoop.fs.FileStatus[]`	`getFileList(List<org.apache.hadoop.fs.Path> logDirs, org.apache.hadoop.fs.PathFilter filter)`
`org.apache.hadoop.hbase.protobuf.generated.ZooKeeperProtos.SplitLogTask.RecoveryMode`	`getRecoveryMode()`
`(package private) ConcurrentMap<String,SplitLogManager.Task>`	`getTasks()`
`(package private) void`	`handleDeadWorker(ServerName workerName)`
`(package private) void`	`handleDeadWorkers(Set<ServerName> serverNames)`
`boolean`	`isLogReplaying()`
`boolean`	`isLogSplitting()`
`void`	`markRegionsRecovering(ServerName server, Set<HRegionInfo> userRegions)`
`private void`	`removeRecoveringRegions(Set<ServerName> serverNames, Boolean isMetaRecovery)` It removes recovering regions under /hbase/recovering-regions/[encoded region name] so that the region server hosting the region can allow reads to the recovered region
`(package private) void`	`removeStaleRecoveringRegions(Set<ServerName> failedServers)` It removes stale recovering regions under /hbase/recovering-regions/[encoded region name] during master initialization phase.
`void`	`setRecoveryMode(boolean isForInitialization)` This function is to set recovery mode from outstanding split log tasks from before or current configuration setting
`long`	`splitLogDistributed(List<org.apache.hadoop.fs.Path> logDirs)` The caller will block until all the log files of the given region server have been processed - successfully split or an error is encountered - by an available worker region server.
`long`	`splitLogDistributed(org.apache.hadoop.fs.Path logDir)`
`long`	`splitLogDistributed(Set<ServerName> serverNames, List<org.apache.hadoop.fs.Path> logDirs, org.apache.hadoop.fs.PathFilter filter)` The caller will block until all the hbase:meta log files of the given region server have been processed - successfully split or an error is encountered - by an available worker region server.
`void`	`stop()`
`private void`	`waitForSplittingCompletion(SplitLogManager.TaskBatch batch, MonitoredTask status)`

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - LOG
```
private static final org.apache.commons.logging.Log LOG
```
  - server
```
private Server server
```
  - stopper
```
private final Stoppable stopper
```
  - conf
```
private final org.apache.hadoop.conf.Configuration conf
```
  - choreService
```
private final ChoreService choreService
```
  - DEFAULT_UNASSIGNED_TIMEOUT
```
public static final int DEFAULT_UNASSIGNED_TIMEOUT
```
    See Also:
    Constant Field Values
  - unassignedTimeout
```
private long unassignedTimeout
```
  - lastTaskCreateTime
```
private long lastTaskCreateTime
```
  - checkRecoveringTimeThreshold
```
private long checkRecoveringTimeThreshold
```
  - failedRecoveringRegionDeletions
```
private final List<Pair<Set<ServerName>,Boolean>> failedRecoveringRegionDeletions
```
  - recoveringRegionLock
```
protected final ReentrantLock recoveringRegionLock
```
    In distributedLogReplay mode, we need touch both splitlog and recovering-regions znodes in one operation. So the lock is used to guard such cases.
  - tasks
```
private final ConcurrentMap<String,SplitLogManager.Task> tasks
```
  - timeoutMonitor
```
private SplitLogManager.TimeoutMonitor timeoutMonitor
```
  - deadWorkers
```
private volatile Set<ServerName> deadWorkers
```
  - deadWorkersLock
```
private final Object deadWorkersLock
```
- Constructor Detail
  - SplitLogManager
```
public SplitLogManager(Server server,
               org.apache.hadoop.conf.Configuration conf,
               Stoppable stopper,
               MasterServices master,
               ServerName serverName)
                throws IOException
```
    Its OK to construct this object even when region-servers are not online. It does lookup the orphan tasks in coordination engine but it doesn't block waiting for them to be done.
    
    Parameters:
    server - the server instance
    conf - the HBase configuration
    stopper - the stoppable in case anything is wrong
    master - the master services
    serverName - the master server name
    
    Throws:
    
    IOException
- Method Detail
  - getFileList
```
private org.apache.hadoop.fs.FileStatus[] getFileList(List<org.apache.hadoop.fs.Path> logDirs,
                                            org.apache.hadoop.fs.PathFilter filter)
                                               throws IOException
```
    Throws:
    
    IOException
  - getFileList
```
public static org.apache.hadoop.fs.FileStatus[] getFileList(org.apache.hadoop.conf.Configuration conf,
                                            List<org.apache.hadoop.fs.Path> logDirs,
                                            org.apache.hadoop.fs.PathFilter filter)
                                                     throws IOException
```
    Get a list of paths that need to be split given a set of server-specific directories and optionally a filter. See DefaultWALProvider.getServerNameFromWALDirectoryName(org.apache.hadoop.conf.Configuration, java.lang.String) for more info on directory layout. Should be package-private, but is needed by WALSplitter.split(Path, Path, Path, FileSystem, Configuration, WALFactory) for tests.
    
    Throws:
    
    IOException
  - splitLogDistributed
```
public long splitLogDistributed(org.apache.hadoop.fs.Path logDir)
                         throws IOException
```
    Parameters:
    logDir - one region sever wal dir path in .logs
    
    Returns:
    cumulative size of the logfiles split
    
    Throws:
    
    IOException - if there was an error while splitting any log file
    
    IOException
  - splitLogDistributed
```
public long splitLogDistributed(List<org.apache.hadoop.fs.Path> logDirs)
                         throws IOException
```
    The caller will block until all the log files of the given region server have been processed - successfully split or an error is encountered - by an available worker region server. This method must only be called after the region servers have been brought online.
    
    Parameters:
    logDirs - List of log dirs to split
    
    Returns:
    cumulative size of the logfiles split
    
    Throws:
    
    IOException - If there was an error while splitting any log file
  - splitLogDistributed
```
public long splitLogDistributed(Set<ServerName> serverNames,
                       List<org.apache.hadoop.fs.Path> logDirs,
                       org.apache.hadoop.fs.PathFilter filter)
                         throws IOException
```
    The caller will block until all the hbase:meta log files of the given region server have been processed - successfully split or an error is encountered - by an available worker region server. This method must only be called after the region servers have been brought online.
    
    Parameters:
    logDirs - List of log dirs to split
    filter - the Path filter to select specific files for considering
    
    Returns:
    cumulative size of the logfiles split
    
    Throws:
    
    IOException - If there was an error while splitting any log file
  - enqueueSplitTask
```
boolean enqueueSplitTask(String taskname,
                       SplitLogManager.TaskBatch batch)
```
    Add a task entry to coordination if it is not already there.
    
    Parameters:
    taskname - the path of the log to be split
    batch - the batch this task belongs to
    
    Returns:
    true if a new entry is created, false if it is already there.
  - waitForSplittingCompletion
```
private void waitForSplittingCompletion(SplitLogManager.TaskBatch batch,
                              MonitoredTask status)
```
  - getTasks
```
ConcurrentMap<String,SplitLogManager.Task> getTasks()
```
  - activeTasks
```
private int activeTasks(SplitLogManager.TaskBatch batch)
```
  - removeRecoveringRegions
```
private void removeRecoveringRegions(Set<ServerName> serverNames,
                           Boolean isMetaRecovery)
```
    It removes recovering regions under /hbase/recovering-regions/[encoded region name] so that the region server hosting the region can allow reads to the recovered region
    
    Parameters:
    serverNames - servers which are just recovered
    isMetaRecovery - whether current recovery is for the meta region on serverNames
  removeStaleRecoveringRegions void removeStaleRecoveringRegions(Set<ServerName> failedServers) throws IOException, InterruptedIOException It removes stale recovering regions under /hbase/recovering-regions/[encoded region name] during master initialization phase. Parameters: failedServers - A set of known failed servers Throws: IOException InterruptedIOException createTaskIfAbsent private SplitLogManager.Task createTaskIfAbsent(String path, SplitLogManager.TaskBatch batch) Parameters: path - batch - Returns: null on success, existing task on error findOrCreateOrphanTask SplitLogManager.Task findOrCreateOrphanTask(String path) stop public void stop() handleDeadWorker void handleDeadWorker(ServerName workerName) handleDeadWorkers void handleDeadWorkers(Set<ServerName> serverNames) setRecoveryMode public void setRecoveryMode(boolean isForInitialization) throws IOException This function is to set recovery mode from outstanding split log tasks from before or current configuration setting Parameters: isForInitialization - Throws: IOException - throws if it's impossible to set recovery mode markRegionsRecovering public void markRegionsRecovering(ServerName server, Set<HRegionInfo> userRegions) throws InterruptedIOException, IOException Throws: InterruptedIOException IOException isLogReplaying public boolean isLogReplaying() Returns: whether log is replaying isLogSplitting public boolean isLogSplitting() Returns: whether log is splitting getRecoveryMode public org.apache.hadoop.hbase.protobuf.generated.ZooKeeperProtos.SplitLogTask.RecoveryMode getRecoveryMode() Returns: the current log recovery mode

Class SplitLogManager

Nested Class Summary

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

LOG

server

stopper

conf

choreService

DEFAULT_UNASSIGNED_TIMEOUT

unassignedTimeout

lastTaskCreateTime

checkRecoveringTimeThreshold

failedRecoveringRegionDeletions

recoveringRegionLock

tasks

timeoutMonitor

deadWorkers

deadWorkersLock

Constructor Detail

SplitLogManager

Method Detail

getFileList

getFileList

splitLogDistributed

splitLogDistributed

splitLogDistributed

enqueueSplitTask

waitForSplittingCompletion

getTasks

activeTasks

removeRecoveringRegions

removeStaleRecoveringRegions

createTaskIfAbsent

findOrCreateOrphanTask

stop

handleDeadWorker

handleDeadWorkers

setRecoveryMode

markRegionsRecovering

isLogReplaying

isLogSplitting

getRecoveryMode