org.apache.hadoop.hbase.master

Class SplitWALManager

java.lang.Object

org.apache.hadoop.hbase.master.SplitWALManager

@InterfaceAudience.Private
public class SplitWALManager
extends Object

Create SplitWALProcedure for each WAL which need to split. Manage the workers for each SplitWALProcedure. Total number of workers is (number of online servers) * (HBASE_SPLIT_WAL_MAX_SPLITTER). Helps assign and release workers for split tasks. Provide helper method to delete split WAL file and directory. The user can get the SplitWALProcedures via splitWALs(crashedServer, splitMeta) can get the files that need to split via getWALsToSplit(crashedServer, splitMeta) can delete the splitting WAL and directory via deleteSplitWAL(wal) and deleteSplitWAL(crashedServer) can check if splitting WALs of a crashed server is success via isSplitWALFinished(walPath) can acquire and release a worker for splitting WAL via acquireSplitWALWorker(procedure) and releaseSplitWALWorker(worker, scheduler) This class is to replace the zk-based WAL splitting related code, MasterWalManager, SplitLogManager, ZKSplitLog and ZKSplitLogManagerCoordination can be removed after we switch to procedure-based WAL splitting.

See Also:

for the original distributed split WAL manager.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
private static class	SplitWALManager.SplitWorkerAssigner help assign and release a worker for each WAL splitting task For each worker, concurrent running splitting task should be no more than maxSplitTasks If a task failed to acquire a worker, it will suspend and wait for workers available

Field Summary

Fields

Modifier and Type	Field and Description
private org.apache.hadoop.conf.Configuration	conf
private org.apache.hadoop.fs.FileSystem	fs
private static org.slf4j.Logger	LOG
private MasterServices	master
private org.apache.hadoop.fs.Path	rootDir
private SplitWALManager.SplitWorkerAssigner	splitWorkerAssigner
private org.apache.hadoop.fs.Path	walArchiveDir

Constructor Summary

Constructors

Constructor and Description
SplitWALManager(MasterServices master)

Method Summary

Modifier and Type	Method and Description
ServerName	acquireSplitWALWorker(Procedure<?> procedure) Acquire a split WAL worker
void	addUsedSplitWALWorker(ServerName worker) When master restart, there will be a new splitWorkerAssigner.
void	archive(String wal) Archive processed WAL
(package private) List<Procedure>	createSplitWALProcedures(List<org.apache.hadoop.fs.FileStatus> splittingWALs, ServerName crashedServer)
void	deleteWALDir(ServerName serverName)
private org.apache.hadoop.fs.Path	getWALSplitDir(ServerName serverName)
List<org.apache.hadoop.fs.FileStatus>	getWALsToSplit(ServerName serverName, boolean splitMeta)
boolean	isSplitWALFinished(String walPath)
void	releaseSplitWALWorker(ServerName worker, MasterProcedureScheduler scheduler) After the worker finished the split WAL task, it will release the worker, and wake up all the suspend procedures in the ProcedureEvent
List<Procedure>	splitWALs(ServerName crashedServer, boolean splitMeta)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

private static final org.slf4j.Logger LOG

master

private final MasterServices master

splitWorkerAssigner

private final SplitWALManager.SplitWorkerAssigner splitWorkerAssigner

rootDir

private final org.apache.hadoop.fs.Path rootDir

fs

private final org.apache.hadoop.fs.FileSystem fs

conf

private final org.apache.hadoop.conf.Configuration conf

walArchiveDir

private final org.apache.hadoop.fs.Path walArchiveDir

Constructor Detail

SplitWALManager

public SplitWALManager(MasterServices master)
                throws IOException

Throws:

IOException

Method Detail

splitWALs

public List<Procedure> splitWALs(ServerName crashedServer,
                                 boolean splitMeta)
                          throws IOException

Throws:

IOException

getWALsToSplit

public List<org.apache.hadoop.fs.FileStatus> getWALsToSplit(ServerName serverName,
                                                            boolean splitMeta)
                                                     throws IOException

Throws:

IOException

getWALSplitDir

private org.apache.hadoop.fs.Path getWALSplitDir(ServerName serverName)

archive

public void archive(String wal)
             throws IOException

Archive processed WAL

Throws:

IOException

deleteWALDir

public void deleteWALDir(ServerName serverName)
                  throws IOException

Throws:

IOException

isSplitWALFinished

public boolean isSplitWALFinished(String walPath)
                           throws IOException

Throws:

IOException

createSplitWALProcedures

List<Procedure> createSplitWALProcedures(List<org.apache.hadoop.fs.FileStatus> splittingWALs,
                                         ServerName crashedServer)

acquireSplitWALWorker

public ServerName acquireSplitWALWorker(Procedure<?> procedure)
                                 throws ProcedureSuspendedException

Acquire a split WAL worker

Parameters:

procedure - split WAL task

Returns:

an available region server which could execute this task

Throws:

ProcedureSuspendedException - if there is no available worker, it will throw this exception to WAIT the procedure.

releaseSplitWALWorker

public void releaseSplitWALWorker(ServerName worker,
                                  MasterProcedureScheduler scheduler)

After the worker finished the split WAL task, it will release the worker, and wake up all the suspend procedures in the ProcedureEvent

Parameters:

worker - worker which is about to release

scheduler - scheduler which is to wake up the procedure event

addUsedSplitWALWorker

public void addUsedSplitWALWorker(ServerName worker)

When master restart, there will be a new splitWorkerAssigner. But if there are splitting WAL tasks running on the region server side, they will not be count by the new splitWorkerAssigner. Thus we should add the workers of running tasks to the assigner when we load the procedures from MasterProcWALs.

Parameters:

worker - region server which is executing a split WAL task