org.apache.hadoop.hbase.coordination

Class ZkSplitLogWorkerCoordination

java.lang.Object

org.apache.hadoop.hbase.zookeeper.ZKListener

org.apache.hadoop.hbase.coordination.ZkSplitLogWorkerCoordination

All Implemented Interfaces:

SplitLogWorkerCoordination

@InterfaceAudience.Private
public class ZkSplitLogWorkerCoordination
extends ZKListener
implements SplitLogWorkerCoordination

ZooKeeper based implementation of SplitLogWorkerCoordination It listen for changes in ZooKeeper and

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
(package private) class	ZkSplitLogWorkerCoordination.GetDataAsyncCallback Asynchronous handler for zk get-data-set-watch on node results.
static class	ZkSplitLogWorkerCoordination.ZkSplitTaskDetails When ZK-based implementation wants to complete the task, it needs to know task znode and current znode cversion (needed for subsequent update operation).

Nested classes/interfaces inherited from interface org.apache.hadoop.hbase.coordination.SplitLogWorkerCoordination

SplitLogWorkerCoordination.SplitTaskDetails

Field Summary

Fields

Modifier and Type	Field and Description
private static int	checkInterval
private String	currentTask
private int	currentVersion
private static int	FAILED_TO_OWN_TASK
private Object	grabTaskLock
private static org.slf4j.Logger	LOG
private int	maxConcurrentTasks
private int	reportPeriod
private RegionServerServices	server
private ServerName	serverName
private boolean	shouldStop
private SplitLogWorker.TaskExecutor	splitTaskExecutor
private AtomicInteger	taskReadySeq
protected AtomicInteger	tasksInProgress
private SplitLogWorker	worker
private boolean	workerInGrabTask

Fields inherited from class org.apache.hadoop.hbase.zookeeper.ZKListener

watcher

Fields inherited from interface org.apache.hadoop.hbase.coordination.SplitLogWorkerCoordination

DEFAULT_MAX_SPLITTERS

Constructor Summary

Constructors

Constructor and Description
ZkSplitLogWorkerCoordination(ServerName serverName, ZKWatcher watcher)

Method Summary

Modifier and Type	Method and Description
private boolean	areSplittersAvailable(int expectedTasksPerRS)
protected static int	attemptToOwnTask(boolean isFirstTime, ZKWatcher zkw, ServerName server, String task, int taskZKVersion) Try to own the task by transitioning the zk node data from UNASSIGNED to OWNED.
void	endTask(SplitLogTask slt, LongAdder ctr, SplitLogWorkerCoordination.SplitTaskDetails details) endTask() can fail and the only way to recover out of it is for the SplitLogManager to timeout the task node.
void	getDataSetWatchAsync()
(package private) void	getDataSetWatchFailure(String path)
(package private) void	getDataSetWatchSuccess(String path, byte[] data)
private int	getNumExpectedTasksPerRS(int numTasks) This function calculates how many splitters this RS should create based on expected average tasks per RS and the hard limit upper bound(maxConcurrentTasks) set by configuration.
private List<String>	getTaskList()
int	getTaskReadySeq() Used by unit tests to check how many tasks were processed
private boolean	grabTask(String path) try to grab a 'lock' on the task zk node to own and execute the task.
void	init(RegionServerServices server, org.apache.hadoop.conf.Configuration conf, SplitLogWorker.TaskExecutor splitExecutor, SplitLogWorker worker) Override setter from SplitLogWorkerCoordination
boolean	isReady() Check whether the log splitter is ready to supply tasks
boolean	isStop()
void	markCorrupted(org.apache.hadoop.fs.Path rootDir, String name, org.apache.hadoop.fs.FileSystem fs) marks log file as corrupted
void	nodeChildrenChanged(String path) Override handler from ZKListener
void	nodeDataChanged(String path) Override handler from ZKListener
void	registerListener() set the listener for task changes.
void	removeListener() remove the listener for task changes.
void	stopProcessingTasks() called when Coordination should stop processing tasks and exit
(package private) void	submitTask(String curTask, int curTaskZKVersion, int reportPeriod) Submit a log split task to executor service
void	taskLoop() Wait for tasks to become available at /hbase/splitlog zknode.

Methods inherited from class org.apache.hadoop.hbase.zookeeper.ZKListener

getWatcher, nodeCreated, nodeDeleted

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

private static final org.slf4j.Logger LOG

checkInterval

private static final int checkInterval

See Also:

Constant Field Values

FAILED_TO_OWN_TASK

private static final int FAILED_TO_OWN_TASK

See Also:

Constant Field Values

worker

private SplitLogWorker worker

splitTaskExecutor

private SplitLogWorker.TaskExecutor splitTaskExecutor

taskReadySeq

private final AtomicInteger taskReadySeq

currentTask

private volatile String currentTask

currentVersion

private int currentVersion

shouldStop

private volatile boolean shouldStop

grabTaskLock

private final Object grabTaskLock

workerInGrabTask

private boolean workerInGrabTask

reportPeriod

private int reportPeriod

server

private RegionServerServices server

tasksInProgress

protected final AtomicInteger tasksInProgress

maxConcurrentTasks

private int maxConcurrentTasks

serverName

private final ServerName serverName

Constructor Detail

ZkSplitLogWorkerCoordination

public ZkSplitLogWorkerCoordination(ServerName serverName,
                                    ZKWatcher watcher)

Method Detail

nodeChildrenChanged

public void nodeChildrenChanged(String path)

Override handler from ZKListener

Overrides:

nodeChildrenChanged in class ZKListener

Parameters:

path - full path of the node whose children have changed

nodeDataChanged

public void nodeDataChanged(String path)

Override handler from ZKListener

Overrides:

nodeDataChanged in class ZKListener

Parameters:

path - full path of the updated node

init

public void init(RegionServerServices server,
                 org.apache.hadoop.conf.Configuration conf,
                 SplitLogWorker.TaskExecutor splitExecutor,
                 SplitLogWorker worker)

Override setter from SplitLogWorkerCoordination

Specified by:

init in interface SplitLogWorkerCoordination

Parameters:

server - instance of RegionServerServices to work with

conf - is current configuration.

splitExecutor - split executor from SplitLogWorker

worker - instance of SplitLogWorker

getDataSetWatchFailure

void getDataSetWatchFailure(String path)

getDataSetWatchAsync

public void getDataSetWatchAsync()

getDataSetWatchSuccess

void getDataSetWatchSuccess(String path,
                            byte[] data)

grabTask

private boolean grabTask(String path)

try to grab a 'lock' on the task zk node to own and execute the task.

Parameters:

path - zk node for the task

Returns:

boolean value when grab a task success return true otherwise false

submitTask

void submitTask(String curTask,
                int curTaskZKVersion,
                int reportPeriod)

Submit a log split task to executor service

Parameters:

curTask - task to submit

curTaskZKVersion - current version of task

getNumExpectedTasksPerRS

private int getNumExpectedTasksPerRS(int numTasks)

This function calculates how many splitters this RS should create based on expected average tasks per RS and the hard limit upper bound(maxConcurrentTasks) set by configuration.
At any given time, a RS allows spawn MIN(Expected Tasks/RS, Hard Upper Bound)

Parameters:

numTasks - total number of split tasks available

Returns:

number of tasks this RS can grab

areSplittersAvailable

private boolean areSplittersAvailable(int expectedTasksPerRS)

Parameters:

expectedTasksPerRS - Average number of tasks to be handled by each RS

Returns:

true if more splitters are available, otherwise false.

attemptToOwnTask

protected static int attemptToOwnTask(boolean isFirstTime,
                                      ZKWatcher zkw,
                                      ServerName server,
                                      String task,
                                      int taskZKVersion)

Try to own the task by transitioning the zk node data from UNASSIGNED to OWNED.

This method is also used to periodically heartbeat the task progress by transitioning the node from OWNED to OWNED.

Parameters:

isFirstTime - shows whther it's the first attempt.

zkw - zk wathcer

server - name

task - to own

taskZKVersion - version of the task in zk

Returns:

non-negative integer value when task can be owned by current region server otherwise -1

taskLoop

public void taskLoop()
              throws InterruptedException

Wait for tasks to become available at /hbase/splitlog zknode. Grab a task one at a time. This policy puts an upper-limit on the number of simultaneous log splitting that could be happening in a cluster.

Synchronization using taskReadySeq ensures that it will try to grab every task that has been put up

Specified by:

taskLoop in interface SplitLogWorkerCoordination

Throws:

InterruptedException

getTaskList

private List<String> getTaskList()
                          throws InterruptedException

Throws:

InterruptedException

markCorrupted

public void markCorrupted(org.apache.hadoop.fs.Path rootDir,
                          String name,
                          org.apache.hadoop.fs.FileSystem fs)

Description copied from interface: SplitLogWorkerCoordination

marks log file as corrupted

Specified by:

markCorrupted in interface SplitLogWorkerCoordination

Parameters:

rootDir - where to find the log

name - of the log

fs - file system

isReady

public boolean isReady()
                throws InterruptedException

Description copied from interface: SplitLogWorkerCoordination

Check whether the log splitter is ready to supply tasks

Specified by:

isReady in interface SplitLogWorkerCoordination

Returns:

false if there is no tasks

Throws:

InterruptedException - if the SplitLogWorker was stopped

getTaskReadySeq

public int getTaskReadySeq()

Description copied from interface: SplitLogWorkerCoordination

Used by unit tests to check how many tasks were processed

Specified by:

getTaskReadySeq in interface SplitLogWorkerCoordination

Returns:

number of tasks

registerListener

public void registerListener()

Description copied from interface: SplitLogWorkerCoordination

set the listener for task changes. Implementation specific

Specified by:

registerListener in interface SplitLogWorkerCoordination

removeListener

public void removeListener()

Description copied from interface: SplitLogWorkerCoordination

remove the listener for task changes. Implementation specific

Specified by:

removeListener in interface SplitLogWorkerCoordination

stopProcessingTasks

public void stopProcessingTasks()

Description copied from interface: SplitLogWorkerCoordination

called when Coordination should stop processing tasks and exit

Specified by:

stopProcessingTasks in interface SplitLogWorkerCoordination

isStop

public boolean isStop()

Specified by:

isStop in interface SplitLogWorkerCoordination

Returns:

the current value of exitWorker

endTask

public void endTask(SplitLogTask slt,
                    LongAdder ctr,
                    SplitLogWorkerCoordination.SplitTaskDetails details)

endTask() can fail and the only way to recover out of it is for the SplitLogManager to timeout the task node.

Specified by:

endTask in interface SplitLogWorkerCoordination

Parameters:

slt -

ctr -

details - details about log split task (specific to coordination engine being used).