Package org.apache.hadoop.hbase.master.procedure

Class MasterProcedureScheduler

java.lang.Object

org.apache.hadoop.hbase.procedure2.AbstractProcedureScheduler

org.apache.hadoop.hbase.master.procedure.MasterProcedureScheduler

All Implemented Interfaces:

ProcedureScheduler

@Private
public class MasterProcedureScheduler
extends AbstractProcedureScheduler

ProcedureScheduler for the Master Procedures. This ProcedureScheduler tries to provide to the ProcedureExecutor procedures that can be executed without having to wait on a lock. Most of the master operations can be executed concurrently, if they are operating on different tables (e.g. two create table procedures can be performed at the same time) or against two different servers; say two servers that crashed at about the same time.

Each procedure should implement an Interface providing information for this queue. For example table related procedures should implement TableProcedureInterface. Each procedure will be pushed in its own queue, and based on the operation type we may make smarter decisions: e.g. we can abort all the operations preceding a delete table, or similar.

Concurrency control

Concurrent access to member variables (tableRunQueue, serverRunQueue, locking, tableMap, serverBuckets) is controlled by schedLock(). This mainly includes:

• AbstractProcedureScheduler.push(Procedure, boolean, boolean): A push will add a Queue back to run-queue when:
  1. Queue was empty before push (so must have been out of run-queue)
  2. Child procedure is added (which means parent procedure holds exclusive lock, and it must have moved Queue out of run-queue)
• AbstractProcedureScheduler.poll(long): A poll will remove a Queue from run-queue when:
  1. Queue becomes empty after poll
  2. Exclusive lock is requested by polled procedure and lock is available (returns the procedure)
  3. Exclusive lock is requested but lock is not available (returns null)
  4. Polled procedure is child of parent holding exclusive lock and the next procedure is not a child
• Namespace/table/region locks: Queue is added back to run-queue when lock being released is:
  1. Exclusive lock
  2. Last shared lock (in case queue was removed because next procedure in queue required exclusive lock)

Field Summary

Fields

Modifier and Type	Field	Description
private static final AvlUtil.AvlKeyComparator<GlobalQueue>	GLOBAL_QUEUE_KEY_COMPARATOR
private GlobalQueue	globalMap
private final FairQueue<String>	globalRunQueue
private final SchemaLocking	locking
private static final org.slf4j.Logger	LOG
private static final AvlUtil.AvlKeyComparator<MetaQueue>	META_QUEUE_KEY_COMPARATOR
private MetaQueue	metaMap
private final FairQueue<TableName>	metaRunQueue
private static final AvlUtil.AvlKeyComparator<PeerQueue>	PEER_QUEUE_KEY_COMPARATOR
private PeerQueue	peerMap
private final FairQueue<String>	peerRunQueue
private final Function<Long,Procedure<?>>	procedureRetriever
private static final AvlUtil.AvlKeyComparator<ServerQueue>	SERVER_QUEUE_KEY_COMPARATOR
private final ServerQueue[]	serverBuckets
private final FairQueue<ServerName>	serverRunQueue
private static final AvlUtil.AvlKeyComparator<TableQueue>	TABLE_QUEUE_KEY_COMPARATOR
private TableQueue	tableMap
private final Map<TableName,TableProcedureWaitingQueue>	tableProcsWaitingEnqueue
private final FairQueue<TableName>	tableRunQueue

Constructor Summary

Constructors

Constructor	Description
MasterProcedureScheduler(Function<Long,Procedure<?>> procedureRetriever)

Method Summary

Modifier and Type	Method	Description
private static <T extends Comparable<T>> void	addToRunQueue(FairQueue<T> fairq, Queue<T> queue, Supplier<String> reason)
void	clear()	Clear current state of scheduler such that it is equivalent to newly created scheduler.
private <T extends Comparable<T>, TNode extends Queue<T>> void	clear(TNode treeMap, FairQueue<T> fairq, AvlUtil.AvlKeyComparator<TNode> comparator)
private void	clearQueue()
void	completionCleanup(Procedure proc)	The procedure in execution completed.
protected Procedure	dequeue()	Fetch one Procedure from the queue NOTE: this method is called with the sched lock held.
private <T extends Comparable<T>> void	doAdd(FairQueue<T> fairq, Queue<T> queue, Procedure<?> proc, boolean addFront)
private <T extends Comparable<T>> Procedure<?>	doPoll(FairQueue<T> fairq)
String	dumpLocks()	For debugging.
protected void	enqueue(Procedure proc, boolean addFront)	Add the procedure to the queue.
private static int	getBucketIndex(Object[] buckets, int hashCode)
private static String	getGlobalId(Procedure<?> proc)
private GlobalQueue	getGlobalQueue(String globalId)
LockedResource	getLockResource(LockedResourceType resourceType, String resourceName)
List<LockedResource>	getLocks()	List lock queues.
private MetaQueue	getMetaQueue()
private static String	getPeerId(Procedure<?> proc)
private PeerQueue	getPeerQueue(String peerId)
private static ServerName	getServerName(Procedure<?> proc)
private ServerQueue	getServerQueue(ServerName serverName, ServerProcedureInterface proc)
private static TableName	getTableName(Procedure<?> proc)
private TableQueue	getTableQueue(TableName tableName)
private static boolean	isGlobalProcedure(Procedure<?> proc)
private <T extends Comparable<T>> boolean	isLockReady(Procedure<?> proc, Queue<T> rq)
private static boolean	isMetaProcedure(Procedure<?> proc)
private static boolean	isPeerProcedure(Procedure<?> proc)
private static boolean	isServerProcedure(Procedure<?> proc)
private static boolean	isTableProcedure(Procedure<?> proc)
private void	logLockedResource(LockedResourceType resourceType, String resourceName)	Get lock info for a resource of specified type and name and log details
(package private) boolean	markTableAsDeleted(TableName table, Procedure<?> procedure)	Tries to remove the queue and the table-lock of the specified table.
protected boolean	queueHasRunnables()	Returns true if there are procedures available to process.
protected int	queueSize()	Returns the number of elements in this queue.
private int	queueSize(Queue<?> head)
private static <T extends Comparable<T>> void	removeFromRunQueue(FairQueue<T> fairq, Queue<T> queue, Supplier<String> reason)
private void	removeGlobalQueue(String globalId)
private void	removePeerQueue(String peerId)
private void	removeServerQueue(ServerName serverName)
private void	removeTableQueue(TableName tableName)
private void	serverBucketToString(org.apache.commons.lang3.builder.ToStringBuilder builder, String queueName, Queue<?> queue)
private boolean	shouldWaitBeforeEnqueuing(TableProcedureInterface proc)
String	toString()
private void	tryCleanupGlobalQueue(String globalId, Procedure<?> procedure)
private void	tryCleanupPeerQueue(String peerId, Procedure<?> procedure)
private <T extends Comparable<T>, TNode extends Queue<T>> boolean	tryCleanupQueue(T id, Procedure<?> proc, Supplier<TNode> getMap, AvlUtil.AvlKeyComparator<TNode> comparator, Function<T,LockAndQueue> getLock, FairQueue<T> runQueue, Consumer<T> removeQueue)
private void	tryCleanupServerQueue(ServerName serverName, Procedure<?> proc)
boolean	waitGlobalExclusiveLock(Procedure<?> procedure, String globalId)	Try to acquire the share lock on global.
boolean	waitMetaExclusiveLock(Procedure<?> procedure)	Deprecated. only used for RecoverMetaProcedure.
boolean	waitNamespaceExclusiveLock(Procedure<?> procedure, String namespace)	Suspend the procedure if the specified namespace is already locked.
boolean	waitPeerExclusiveLock(Procedure<?> procedure, String peerId)	Try to acquire the exclusive lock on the specified peer.
boolean	waitRegion(Procedure<?> procedure, RegionInfo regionInfo)	Suspend the procedure if the specified region is already locked.
boolean	waitRegions(Procedure<?> procedure, TableName table, RegionInfo... regionInfos)	Suspend the procedure if the specified set of regions are already locked.
boolean	waitServerExclusiveLock(Procedure<?> procedure, ServerName serverName)	Try to acquire the exclusive lock on the specified server.
boolean	waitTableExclusiveLock(Procedure<?> procedure, TableName table)	Suspend the procedure if the specified table is already locked.
private TableQueue	waitTableQueueSharedLock(Procedure<?> procedure, TableName table)
boolean	waitTableSharedLock(Procedure<?> procedure, TableName table)	Suspend the procedure if the specified table is already locked.
void	wakeGlobalExclusiveLock(Procedure<?> procedure, String globalId)	Wake the procedures waiting for global.
void	wakeMetaExclusiveLock(Procedure<?> procedure)	Deprecated. only used for RecoverMetaProcedure.
void	wakeNamespaceExclusiveLock(Procedure<?> procedure, String namespace)	Wake the procedures waiting for the specified namespace
void	wakePeerExclusiveLock(Procedure<?> procedure, String peerId)	Wake the procedures waiting for the specified peer
void	wakeRegion(Procedure<?> procedure, RegionInfo regionInfo)	Wake the procedures waiting for the specified region
void	wakeRegions(Procedure<?> procedure, TableName table, RegionInfo... regionInfos)	Wake the procedures waiting for the specified regions
void	wakeServerExclusiveLock(Procedure<?> procedure, ServerName serverName)	Wake the procedures waiting for the specified server
void	wakeTableExclusiveLock(Procedure<?> procedure, TableName table)	Wake the procedures waiting for the specified table
void	wakeTableSharedLock(Procedure<?> procedure, TableName table)	Wake the procedures waiting for the specified table
void	yield(Procedure proc)	The procedure can't run at the moment.

Methods inherited from class org.apache.hadoop.hbase.procedure2.AbstractProcedureScheduler

addBack, addBack, addFront, addFront, addFront, getNullPollCalls, getPollCalls, hasRunnables, poll, poll, poll, push, schedLock, schedUnlock, signalAll, size, start, stop, waitProcedure, wakeEvents, wakePollIfNeeded, wakeProcedure, wakeWaitingProcedures

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Details

LOG

private static final org.slf4j.Logger LOG

SERVER_QUEUE_KEY_COMPARATOR

private static final AvlUtil.AvlKeyComparator<ServerQueue> SERVER_QUEUE_KEY_COMPARATOR

TABLE_QUEUE_KEY_COMPARATOR

private static final AvlUtil.AvlKeyComparator<TableQueue> TABLE_QUEUE_KEY_COMPARATOR

PEER_QUEUE_KEY_COMPARATOR

private static final AvlUtil.AvlKeyComparator<PeerQueue> PEER_QUEUE_KEY_COMPARATOR

META_QUEUE_KEY_COMPARATOR

private static final AvlUtil.AvlKeyComparator<MetaQueue> META_QUEUE_KEY_COMPARATOR

GLOBAL_QUEUE_KEY_COMPARATOR

private static final AvlUtil.AvlKeyComparator<GlobalQueue> GLOBAL_QUEUE_KEY_COMPARATOR

serverRunQueue

private final FairQueue<ServerName> serverRunQueue

tableRunQueue

private final FairQueue<TableName> tableRunQueue

peerRunQueue

private final FairQueue<String> peerRunQueue

metaRunQueue

private final FairQueue<TableName> metaRunQueue

globalRunQueue

private final FairQueue<String> globalRunQueue

serverBuckets

private final ServerQueue[] serverBuckets

tableMap

private TableQueue tableMap

peerMap

private PeerQueue peerMap

metaMap

private MetaQueue metaMap

globalMap

private GlobalQueue globalMap

procedureRetriever

private final Function<Long,Procedure<?>> procedureRetriever

locking

private final SchemaLocking locking

tableProcsWaitingEnqueue

private final Map<TableName,TableProcedureWaitingQueue> tableProcsWaitingEnqueue

Constructor Details

MasterProcedureScheduler

public MasterProcedureScheduler(Function<Long,Procedure<?>> procedureRetriever)

Method Details

yield

public void yield(Procedure proc)

Description copied from interface: ProcedureScheduler

The procedure can't run at the moment. add it back to the queue, giving priority to someone else.

Parameters:

proc - the Procedure to add back to the list

shouldWaitBeforeEnqueuing

private boolean shouldWaitBeforeEnqueuing(TableProcedureInterface proc)

enqueue

protected void enqueue(Procedure proc,
 boolean addFront)

Description copied from class: AbstractProcedureScheduler

Add the procedure to the queue. NOTE: this method is called with the sched lock held.

Specified by:

enqueue in class AbstractProcedureScheduler

Parameters:

proc - the Procedure to add

addFront - true if the item should be added to the front of the queue

doAdd

private <T extends Comparable<T>> void doAdd(FairQueue<T> fairq,
 Queue<T> queue,
 Procedure<?> proc,
 boolean addFront)

queueHasRunnables

protected boolean queueHasRunnables()

Description copied from class: AbstractProcedureScheduler

Returns true if there are procedures available to process. NOTE: this method is called with the sched lock held.

Specified by:

queueHasRunnables in class AbstractProcedureScheduler

Returns:

true if there are procedures available to process, otherwise false.

dequeue

protected Procedure dequeue()

Description copied from class: AbstractProcedureScheduler

Fetch one Procedure from the queue NOTE: this method is called with the sched lock held.

Specified by:

dequeue in class AbstractProcedureScheduler

Returns:

the Procedure to execute, or null if nothing is available.

isLockReady

private <T extends Comparable<T>> boolean isLockReady(Procedure<?> proc,
 Queue<T> rq)

doPoll

private <T extends Comparable<T>> Procedure<?> doPoll(FairQueue<T> fairq)

getLocks

public List<LockedResource> getLocks()

Description copied from interface: ProcedureScheduler

List lock queues.

Returns:

the locks

getLockResource

public LockedResource getLockResource(LockedResourceType resourceType,
 String resourceName)

Returns:

LockedResource for resource of specified type & name. null if resource is not locked.

clear

public void clear()

Description copied from interface: ProcedureScheduler

Clear current state of scheduler such that it is equivalent to newly created scheduler. Used for testing failure and recovery. To emulate server crash/restart, ProcedureExecutor resets its own state and calls clear() on scheduler.

clearQueue

private void clearQueue()

clear

private <T extends Comparable<T>,
TNode extends Queue<T>>
void clear(TNode treeMap,
 FairQueue<T> fairq,
 AvlUtil.AvlKeyComparator<TNode> comparator)

queueSize

private int queueSize(Queue<?> head)

queueSize

protected int queueSize()

Description copied from class: AbstractProcedureScheduler

Returns the number of elements in this queue. NOTE: this method is called with the sched lock held.

Specified by:

queueSize in class AbstractProcedureScheduler

Returns:

the number of elements in this queue.

completionCleanup

public void completionCleanup(Procedure proc)

Description copied from interface: ProcedureScheduler

The procedure in execution completed. This can be implemented to perform cleanups.

Parameters:

proc - the Procedure that completed the execution.

addToRunQueue

private static <T extends Comparable<T>> void addToRunQueue(FairQueue<T> fairq,
 Queue<T> queue,
 Supplier<String> reason)

removeFromRunQueue

private static <T extends Comparable<T>> void removeFromRunQueue(FairQueue<T> fairq,
 Queue<T> queue,
 Supplier<String> reason)

getTableQueue

private TableQueue getTableQueue(TableName tableName)

removeTableQueue

private void removeTableQueue(TableName tableName)

markTableAsDeleted

boolean markTableAsDeleted(TableName table,
 Procedure<?> procedure)

Tries to remove the queue and the table-lock of the specified table. If there are new operations pending (e.g. a new create), the remove will not be performed.

Parameters:

table - the name of the table that should be marked as deleted

procedure - the procedure that is removing the table

Returns:

true if deletion succeeded, false otherwise meaning that there are other new operations pending for that table (e.g. a new create).

isTableProcedure

private static boolean isTableProcedure(Procedure<?> proc)

getTableName

private static TableName getTableName(Procedure<?> proc)

getServerQueue

private ServerQueue getServerQueue(ServerName serverName,
 ServerProcedureInterface proc)

removeServerQueue

private void removeServerQueue(ServerName serverName)

tryCleanupServerQueue

private void tryCleanupServerQueue(ServerName serverName,
 Procedure<?> proc)

getBucketIndex

private static int getBucketIndex(Object[] buckets,
 int hashCode)

isServerProcedure

private static boolean isServerProcedure(Procedure<?> proc)

getServerName

private static ServerName getServerName(Procedure<?> proc)

getPeerQueue

private PeerQueue getPeerQueue(String peerId)

removePeerQueue

private void removePeerQueue(String peerId)

tryCleanupPeerQueue

private void tryCleanupPeerQueue(String peerId,
 Procedure<?> procedure)

isPeerProcedure

private static boolean isPeerProcedure(Procedure<?> proc)

getPeerId

private static String getPeerId(Procedure<?> proc)

getMetaQueue

private MetaQueue getMetaQueue()

isMetaProcedure

private static boolean isMetaProcedure(Procedure<?> proc)

getGlobalQueue

private GlobalQueue getGlobalQueue(String globalId)

removeGlobalQueue

private void removeGlobalQueue(String globalId)

tryCleanupGlobalQueue

private void tryCleanupGlobalQueue(String globalId,
 Procedure<?> procedure)

isGlobalProcedure

private static boolean isGlobalProcedure(Procedure<?> proc)

getGlobalId

private static String getGlobalId(Procedure<?> proc)

tryCleanupQueue

private <T extends Comparable<T>,
TNode extends Queue<T>>
boolean tryCleanupQueue(T id,
 Procedure<?> proc,
 Supplier<TNode> getMap,
 AvlUtil.AvlKeyComparator<TNode> comparator,
 Function<T,LockAndQueue> getLock,
 FairQueue<T> runQueue,
 Consumer<T> removeQueue)

logLockedResource

private void logLockedResource(LockedResourceType resourceType,
 String resourceName)

Get lock info for a resource of specified type and name and log details

waitTableExclusiveLock

public boolean waitTableExclusiveLock(Procedure<?> procedure,
 TableName table)

Suspend the procedure if the specified table is already locked. Other operations in the table-queue will be executed after the lock is released.

Parameters:

procedure - the procedure trying to acquire the lock

table - Table to lock

Returns:

true if the procedure has to wait for the table to be available

wakeTableExclusiveLock

public void wakeTableExclusiveLock(Procedure<?> procedure,
 TableName table)

Wake the procedures waiting for the specified table

Parameters:

procedure - the procedure releasing the lock

table - the name of the table that has the exclusive lock

waitTableSharedLock

public boolean waitTableSharedLock(Procedure<?> procedure,
 TableName table)

Suspend the procedure if the specified table is already locked. other "read" operations in the table-queue may be executed concurrently,

Parameters:

procedure - the procedure trying to acquire the lock

table - Table to lock

Returns:

true if the procedure has to wait for the table to be available

waitTableQueueSharedLock

private TableQueue waitTableQueueSharedLock(Procedure<?> procedure,
 TableName table)

wakeTableSharedLock

public void wakeTableSharedLock(Procedure<?> procedure,
 TableName table)

Wake the procedures waiting for the specified table

Parameters:

procedure - the procedure releasing the lock

table - the name of the table that has the shared lock

waitRegion

public boolean waitRegion(Procedure<?> procedure,
 RegionInfo regionInfo)

Suspend the procedure if the specified region is already locked.

Parameters:

procedure - the procedure trying to acquire the lock on the region

regionInfo - the region we are trying to lock

Returns:

true if the procedure has to wait for the regions to be available

waitRegions

public boolean waitRegions(Procedure<?> procedure,
 TableName table,
 RegionInfo... regionInfos)

Suspend the procedure if the specified set of regions are already locked.

Parameters:

procedure - the procedure trying to acquire the lock on the regions

table - the table name of the regions we are trying to lock

regionInfos - the list of regions we are trying to lock

Returns:

true if the procedure has to wait for the regions to be available

wakeRegion

public void wakeRegion(Procedure<?> procedure,
 RegionInfo regionInfo)

Wake the procedures waiting for the specified region

Parameters:

procedure - the procedure that was holding the region

regionInfo - the region the procedure was holding

wakeRegions

public void wakeRegions(Procedure<?> procedure,
 TableName table,
 RegionInfo... regionInfos)

Wake the procedures waiting for the specified regions

Parameters:

procedure - the procedure that was holding the regions

regionInfos - the list of regions the procedure was holding

waitNamespaceExclusiveLock

public boolean waitNamespaceExclusiveLock(Procedure<?> procedure,
 String namespace)

Suspend the procedure if the specified namespace is already locked.

Parameters:

procedure - the procedure trying to acquire the lock

namespace - Namespace to lock

Returns:

true if the procedure has to wait for the namespace to be available

See Also:

• wakeNamespaceExclusiveLock(Procedure,String)

wakeNamespaceExclusiveLock

public void wakeNamespaceExclusiveLock(Procedure<?> procedure,
 String namespace)

Wake the procedures waiting for the specified namespace

Parameters:

procedure - the procedure releasing the lock

namespace - the namespace that has the exclusive lock

See Also:

• waitNamespaceExclusiveLock(Procedure,String)

waitServerExclusiveLock

public boolean waitServerExclusiveLock(Procedure<?> procedure,
 ServerName serverName)

Try to acquire the exclusive lock on the specified server.

Parameters:

procedure - the procedure trying to acquire the lock

serverName - Server to lock

Returns:

true if the procedure has to wait for the server to be available

See Also:

• wakeServerExclusiveLock(Procedure,ServerName)

wakeServerExclusiveLock

public void wakeServerExclusiveLock(Procedure<?> procedure,
 ServerName serverName)

Wake the procedures waiting for the specified server

Parameters:

procedure - the procedure releasing the lock

serverName - the server that has the exclusive lock

See Also:

• waitServerExclusiveLock(Procedure,ServerName)

waitPeerExclusiveLock

public boolean waitPeerExclusiveLock(Procedure<?> procedure,
 String peerId)

Try to acquire the exclusive lock on the specified peer.

Parameters:

procedure - the procedure trying to acquire the lock

peerId - peer to lock

Returns:

true if the procedure has to wait for the peer to be available

See Also:

• wakePeerExclusiveLock(Procedure, String)

wakePeerExclusiveLock

public void wakePeerExclusiveLock(Procedure<?> procedure,
 String peerId)

Wake the procedures waiting for the specified peer

Parameters:

procedure - the procedure releasing the lock

peerId - the peer that has the exclusive lock

See Also:

• waitPeerExclusiveLock(Procedure, String)

waitMetaExclusiveLock

@Deprecated
public boolean waitMetaExclusiveLock(Procedure<?> procedure)

Deprecated.

only used for RecoverMetaProcedure. Should be removed along with RecoverMetaProcedure.

Try to acquire the exclusive lock on meta.

Parameters:

procedure - the procedure trying to acquire the lock

Returns:

true if the procedure has to wait for meta to be available

See Also:

• wakeMetaExclusiveLock(Procedure)

wakeMetaExclusiveLock

@Deprecated
public void wakeMetaExclusiveLock(Procedure<?> procedure)

Deprecated.

only used for RecoverMetaProcedure. Should be removed along with RecoverMetaProcedure.

Wake the procedures waiting for meta.

Parameters:

procedure - the procedure releasing the lock

See Also:

• waitMetaExclusiveLock(Procedure)

waitGlobalExclusiveLock

public boolean waitGlobalExclusiveLock(Procedure<?> procedure,
 String globalId)

Try to acquire the share lock on global.

Parameters:

procedure - the procedure trying to acquire the lock

Returns:

true if the procedure has to wait for global to be available

See Also:

• wakeGlobalExclusiveLock(Procedure, String)

wakeGlobalExclusiveLock

public void wakeGlobalExclusiveLock(Procedure<?> procedure,
 String globalId)

Wake the procedures waiting for global.

Parameters:

procedure - the procedure releasing the lock

See Also:

• waitGlobalExclusiveLock(Procedure, String)

dumpLocks

public String dumpLocks()
                 throws IOException

For debugging. Expensive.

Throws:

IOException

serverBucketToString

private void serverBucketToString(org.apache.commons.lang3.builder.ToStringBuilder builder,
 String queueName,
 Queue<?> queue)

toString

public String toString()

Overrides:

toString in class AbstractProcedureScheduler