org.apache.hadoop.hbase.master

Class SplitLogManager

java.lang.Object

org.apache.hadoop.hbase.master.SplitLogManager

Deprecated.

since 2.4.0 and in 3.0.0, to be removed in 4.0.0, replaced by procedure-based distributed WAL splitter, see SplitWALManager.

@Deprecated
 @InterfaceAudience.Private
public class SplitLogManager
extends Object

Distributes the task of log splitting to the available region servers. Coordination happens via coordination engine. For every log file that has to be split a task is created. SplitLogWorkers race to grab a task.

SplitLogManager monitors the tasks that it creates using the timeoutMonitor thread. If a task's progress is slow then SplitLogManagerCoordination.checkTasks() will take away the task from the owner SplitLogWorker and the task will be up for grabs again. When the task is done then it is deleted by SplitLogManager.

Clients call splitLogDistributed(Path) to split a region server's log files. The caller thread waits in this method until all the log files have been split.

All the coordination calls made by this class are asynchronous. This is mainly to help reduce response time seen by the callers.

There is race in this design between the SplitLogManager and the SplitLogWorker. SplitLogManager might re-queue a task that has in reality already been completed by a SplitLogWorker. We rely on the idempotency of the log splitting task for correctness.

It is also assumed that every log splitting task is unique and once completed (either with success or with error) it will be not be submitted again. If a task is resubmitted then there is a risk that old "delete task" can delete the re-submission.

See Also:

for an alternate implementation based on Procedures.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	SplitLogManager.ResubmitDirective Deprecated.
static class	SplitLogManager.Task Deprecated. in memory state of an active task.
static class	SplitLogManager.TaskBatch Deprecated. Keeps track of the batch of tasks submitted together by a caller in splitLogDistributed().
static class	SplitLogManager.TerminationStatus Deprecated.
private class	SplitLogManager.TimeoutMonitor Deprecated. Periodically checks all active tasks and resubmits the ones that have timed out

Field Summary

Fields

Modifier and Type	Field and Description
private ChoreService	choreService Deprecated.
private org.apache.hadoop.conf.Configuration	conf Deprecated.
private Set<ServerName>	deadWorkers Deprecated.
private Object	deadWorkersLock Deprecated.
static int	DEFAULT_UNASSIGNED_TIMEOUT Deprecated.
private long	lastTaskCreateTime Deprecated.
private static org.slf4j.Logger	LOG Deprecated.
private MasterServices	server Deprecated.
(package private) ConcurrentMap<String,SplitLogManager.Task>	tasks Deprecated.
private SplitLogManager.TimeoutMonitor	timeoutMonitor Deprecated.
private long	unassignedTimeout Deprecated.

Constructor Summary

Constructors

Constructor and Description
SplitLogManager(MasterServices master, org.apache.hadoop.conf.Configuration conf) Deprecated. Its OK to construct this object even when region-servers are not online.

Method Summary

Modifier and Type	Method and Description
private int	activeTasks(SplitLogManager.TaskBatch batch) Deprecated.
private SplitLogManager.Task	createTaskIfAbsent(String path, SplitLogManager.TaskBatch batch) Deprecated. nn * @return null on success, existing task on error
(package private) boolean	enqueueSplitTask(String taskname, SplitLogManager.TaskBatch batch) Deprecated. Add a task entry to coordination if it is not already there.
(package private) static int	getBatchWaitTimeMillis(int remainingTasks) Deprecated. Get the amount of time in milliseconds to wait till next check.
static List<org.apache.hadoop.fs.FileStatus>	getFileList(org.apache.hadoop.conf.Configuration conf, List<org.apache.hadoop.fs.Path> logDirs, org.apache.hadoop.fs.PathFilter filter) Deprecated. Get a list of paths that need to be split given a set of server-specific directories and optionally a filter.
private List<org.apache.hadoop.fs.FileStatus>	getFileList(List<org.apache.hadoop.fs.Path> logDirs, org.apache.hadoop.fs.PathFilter filter) Deprecated.
private SplitLogManagerCoordination	getSplitLogManagerCoordination() Deprecated.
(package private) ConcurrentMap<String,SplitLogManager.Task>	getTasks() Deprecated.
(package private) void	handleDeadWorker(ServerName workerName) Deprecated.
(package private) void	handleDeadWorkers(Set<ServerName> serverNames) Deprecated.
long	splitLogDistributed(List<org.apache.hadoop.fs.Path> logDirs) Deprecated. The caller will block until all the log files of the given region server have been processed - successfully split or an error is encountered - by an available worker region server.
long	splitLogDistributed(org.apache.hadoop.fs.Path logDir) Deprecated.
long	splitLogDistributed(Set<ServerName> serverNames, List<org.apache.hadoop.fs.Path> logDirs, org.apache.hadoop.fs.PathFilter filter) Deprecated. The caller will block until all the hbase:meta log files of the given region server have been processed - successfully split or an error is encountered - by an available worker region server.
void	stop() Deprecated.
private void	waitForSplittingCompletion(SplitLogManager.TaskBatch batch, MonitoredTask status) Deprecated.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

private static final org.slf4j.Logger LOG

Deprecated.

server

private final MasterServices server

Deprecated.

conf

private final org.apache.hadoop.conf.Configuration conf

Deprecated.

choreService

private final ChoreService choreService

Deprecated.

DEFAULT_UNASSIGNED_TIMEOUT

public static final int DEFAULT_UNASSIGNED_TIMEOUT

Deprecated.

See Also:

Constant Field Values

unassignedTimeout

private long unassignedTimeout

Deprecated.

lastTaskCreateTime

private long lastTaskCreateTime

Deprecated.

tasks

final ConcurrentMap<String,SplitLogManager.Task> tasks

Deprecated.

timeoutMonitor

private SplitLogManager.TimeoutMonitor timeoutMonitor

Deprecated.

deadWorkers

private volatile Set<ServerName> deadWorkers

Deprecated.

deadWorkersLock

private final Object deadWorkersLock

Deprecated.

Constructor Detail

SplitLogManager

public SplitLogManager(MasterServices master,
                       org.apache.hadoop.conf.Configuration conf)
                throws IOException

Deprecated.

Its OK to construct this object even when region-servers are not online. It does lookup the orphan tasks in coordination engine but it doesn't block waiting for them to be done.

Parameters:

master - the master services

conf - the HBase configuration n

Throws:

IOException

Method Detail

getSplitLogManagerCoordination

private SplitLogManagerCoordination getSplitLogManagerCoordination()

Deprecated.

getFileList

private List<org.apache.hadoop.fs.FileStatus> getFileList(List<org.apache.hadoop.fs.Path> logDirs,
                                                          org.apache.hadoop.fs.PathFilter filter)
                                                   throws IOException

Deprecated.

Throws:

IOException

getFileList

public static List<org.apache.hadoop.fs.FileStatus> getFileList(org.apache.hadoop.conf.Configuration conf,
                                                                List<org.apache.hadoop.fs.Path> logDirs,
                                                                org.apache.hadoop.fs.PathFilter filter)
                                                         throws IOException

Deprecated.

Get a list of paths that need to be split given a set of server-specific directories and optionally a filter.

See AbstractFSWALProvider.getServerNameFromWALDirectoryName(org.apache.hadoop.conf.Configuration, java.lang.String) for more info on directory layout.

Should be package-private, but is needed by WALSplitter.split(Path, Path, Path, FileSystem, Configuration, org.apache.hadoop.hbase.wal.WALFactory) for tests.

Throws:

IOException

splitLogDistributed

public long splitLogDistributed(org.apache.hadoop.fs.Path logDir)
                         throws IOException

Deprecated.

Parameters:

logDir - one region sever wal dir path in .logs

Returns:

cumulative size of the logfiles split n

Throws:

IOException - if there was an error while splitting any log file

splitLogDistributed

public long splitLogDistributed(List<org.apache.hadoop.fs.Path> logDirs)
                         throws IOException

Deprecated.

The caller will block until all the log files of the given region server have been processed - successfully split or an error is encountered - by an available worker region server. This method must only be called after the region servers have been brought online.

Parameters:

logDirs - List of log dirs to split

Returns:

cumulative size of the logfiles split

Throws:

IOException - If there was an error while splitting any log file

splitLogDistributed

public long splitLogDistributed(Set<ServerName> serverNames,
                                List<org.apache.hadoop.fs.Path> logDirs,
                                org.apache.hadoop.fs.PathFilter filter)
                         throws IOException

Deprecated.

The caller will block until all the hbase:meta log files of the given region server have been processed - successfully split or an error is encountered - by an available worker region server. This method must only be called after the region servers have been brought online.

Parameters:

logDirs - List of log dirs to split

filter - the Path filter to select specific files for considering

Returns:

cumulative size of the logfiles split

Throws:

IOException - If there was an error while splitting any log file

enqueueSplitTask

boolean enqueueSplitTask(String taskname,
                         SplitLogManager.TaskBatch batch)

Deprecated.

Add a task entry to coordination if it is not already there.

Parameters:

taskname - the path of the log to be split

batch - the batch this task belongs to

Returns:

true if a new entry is created, false if it is already there.

getBatchWaitTimeMillis

static int getBatchWaitTimeMillis(int remainingTasks)

Deprecated.

Get the amount of time in milliseconds to wait till next check. Check less frequently if a bunch of work to do still. At a max, check every minute. At a minimum, check every 100ms. This is to alleviate case where perhaps there are a bunch of threads waiting on a completion. For example, if the zk-based implementation, we will scan the '/hbase/splitWAL' dir every time through this loop. If there are lots of WALs to split -- could be tens of thousands if big cluster -- then it will take a while. If the Master has many SCPs waiting on wal splitting -- could be up to 10 x the configured PE thread count (default would be 160) -- then the Master will be putting up a bunch of load on zk.

waitForSplittingCompletion

private void waitForSplittingCompletion(SplitLogManager.TaskBatch batch,
                                        MonitoredTask status)

Deprecated.

getTasks

ConcurrentMap<String,SplitLogManager.Task> getTasks()

Deprecated.

activeTasks

private int activeTasks(SplitLogManager.TaskBatch batch)

Deprecated.

createTaskIfAbsent

private SplitLogManager.Task createTaskIfAbsent(String path,
                                                SplitLogManager.TaskBatch batch)

Deprecated.

nn * @return null on success, existing task on error

stop

public void stop()

Deprecated.

handleDeadWorker

void handleDeadWorker(ServerName workerName)

Deprecated.

handleDeadWorkers

void handleDeadWorkers(Set<ServerName> serverNames)

Deprecated.