Package org.apache.hadoop.hbase.master.procedure

Class SnapshotProcedure

java.lang.Object

org.apache.hadoop.hbase.procedure2.Procedure<TEnvironment>

org.apache.hadoop.hbase.procedure2.StateMachineProcedure<MasterProcedureEnv,TState>

org.apache.hadoop.hbase.master.procedure.AbstractStateMachineTableProcedure<org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProcedureProtos.SnapshotState>

org.apache.hadoop.hbase.master.procedure.SnapshotProcedure

All Implemented Interfaces:

Comparable<Procedure<MasterProcedureEnv>>, TableProcedureInterface

@Private
public class SnapshotProcedure
extends AbstractStateMachineTableProcedure<org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProcedureProtos.SnapshotState>

A procedure used to take snapshot on tables.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.hbase.procedure2.StateMachineProcedure

StateMachineProcedure.Flow

Nested classes/interfaces inherited from class org.apache.hadoop.hbase.procedure2.Procedure

Procedure.LockState

Nested classes/interfaces inherited from interface org.apache.hadoop.hbase.master.procedure.TableProcedureInterface

TableProcedureInterface.TableOperationType

Field Summary

Fields

Modifier and Type	Field	Description
private org.apache.hadoop.conf.Configuration	conf
private TableDescriptor	htd
private static final org.slf4j.Logger	LOG
private final MetricsSnapshot	metricsSnapshot
private RetryCounter	retryCounter
private org.apache.hadoop.fs.Path	rootDir
private org.apache.hadoop.fs.FileSystem	rootFs
private org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription	snapshot
private org.apache.hadoop.fs.Path	snapshotDir
private SnapshotManifest	snapshotManifest
private TableName	snapshotTable
private MonitoredTask	status
private org.apache.hadoop.fs.Path	workingDir
private org.apache.hadoop.fs.FileSystem	workingDirFS

Fields inherited from class org.apache.hadoop.hbase.procedure2.StateMachineProcedure

stateCount

Fields inherited from class org.apache.hadoop.hbase.procedure2.Procedure

NO_PROC_ID, NO_TIMEOUT

Constructor Summary

Constructors

Constructor	Description
SnapshotProcedure()
SnapshotProcedure(MasterProcedureEnv env, org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription snapshot)

Method Summary

Modifier and Type	Method	Description
protected Procedure.LockState	acquireLock(MasterProcedureEnv env)	The user should override this method if they need a lock on an Entity.
protected void	afterReplay(MasterProcedureEnv env)	Called when the procedure is ready to be added to the queue after the loading/replay operation.
private void	completeSnapshot(MasterProcedureEnv env)
private Procedure<MasterProcedureEnv>[]	createRemoteSnapshotProcedures(MasterProcedureEnv env)
protected void	deserializeStateData(ProcedureStateSerializer serializer)	Called on store load to allow the user to decode the previously serialized state.
protected StateMachineProcedure.Flow	executeFromState(MasterProcedureEnv env, org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProcedureProtos.SnapshotState state)	called to perform a single step of the specified 'state' of the procedure
private Stream<RegionInfo>	getDefaultRegionReplica(MasterProcedureEnv env)
protected org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProcedureProtos.SnapshotState	getInitialState()	Return the initial state object that will be used for the first call to executeFromState().
org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription	getSnapshot()
org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription	getSnapshotDesc()
protected org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProcedureProtos.SnapshotState	getState(int stateId)	Convert an ordinal (or state id) to an Enum (or more descriptive) state object.
protected int	getStateId(org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProcedureProtos.SnapshotState state)	Convert the Enum (or more descriptive) state object to an ordinal (or state id).
TableName	getTableName()	Returns the name of the table the procedure is operating on
TableProcedureInterface.TableOperationType	getTableOperationType()	Given an operation type we can take decisions about what to do with pending operations.
protected boolean	holdLock(MasterProcedureEnv env)	Used to keep the procedure lock even when the procedure is yielding or suspended.
private boolean	isAnySplitOrMergeProcedureRunning(MasterProcedureEnv env)
protected boolean	isRollbackSupported(org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProcedureProtos.SnapshotState state)	Used by the default implementation of abort() to know if the current state can be aborted and rollback can be triggered.
boolean	isSnapshotCorrupted()
private TableDescriptor	loadTableDescriptorSnapshot(MasterProcedureEnv env)
void	markSnapshotCorrupted()
private void	postSnapshot(MasterProcedureEnv env)
private void	prepareSnapshot(MasterProcedureEnv env)
private void	prepareSnapshotEnv(MasterProcedureEnv env)
private void	preSnapshot(MasterProcedureEnv env)
protected void	releaseLock(MasterProcedureEnv env)	The user should override this method, and release lock if necessary.
protected void	rollbackState(MasterProcedureEnv env, org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProcedureProtos.SnapshotState state)	called to perform the rollback of the specified state
protected void	serializeStateData(ProcedureStateSerializer serializer)	The user-level code of the procedure may have some state to persist (e.g.
protected boolean	setTimeoutFailure(MasterProcedureEnv env)	Called by the ProcedureExecutor when the timeout set by setTimeout() is expired.
private void	snapshotClosedRegions(MasterProcedureEnv env)
private void	snapshotMobRegion(MasterProcedureEnv env)
private void	snapshotSplitOrClosedRegions(MasterProcedureEnv env, List<RegionInfo> regions, String threadPoolName)
private void	snapshotSplitRegions(MasterProcedureEnv env)
void	toStringClassDetails(StringBuilder builder)	Extend the toString() information with the procedure details e.g.
private void	verifySnapshot(MasterProcedureEnv env)

Methods inherited from class org.apache.hadoop.hbase.master.procedure.AbstractStateMachineTableProcedure

checkOnline, checkTableModifiable, getUser, getWALRegionDir, isTableEnabled, preflightChecks, releaseSyncLatch, setUser, waitInitialized

Methods inherited from class org.apache.hadoop.hbase.procedure2.StateMachineProcedure

abort, addChildProcedure, execute, failIfAborted, getCurrentState, getCurrentStateId, getCycles, isEofState, isRollbackSupported, isYieldAfterExecutionStep, isYieldBeforeExecuteFromState, rollback, setNextState, toStringState

Methods inherited from class org.apache.hadoop.hbase.procedure2.Procedure

addStackIndex, beforeReplay, bypass, compareTo, completionCleanup, doExecute, doRollback, elapsedTime, getChildrenLatch, getException, getLastUpdate, getNonceKey, getOwner, getParentProcId, getProcedureMetrics, getProcId, getProcIdHashCode, getProcName, getResult, getRootProcedureId, getRootProcId, getStackIndexes, getState, getSubmittedTime, getTimeout, getTimeoutTimestamp, hasChildren, hasException, hasLock, hasOwner, hasParent, hasTimeout, haveSameParent, incChildrenLatch, isBypass, isFailed, isFinished, isInitializing, isLockedWhenLoading, isRunnable, isSuccess, isWaiting, removeStackIndex, setAbortFailure, setChildrenLatch, setExecuted, setFailure, setFailure, setLastUpdate, setNonceKey, setOwner, setOwner, setParentProcId, setProcId, setResult, setRootProcId, setStackIndexes, setState, setSubmittedTime, setTimeout, shouldWaitClientAck, skipPersistence, suspend, toString, toStringClass, toStringDetails, toStringSimpleSB, updateMetricsOnFinish, updateMetricsOnSubmit, updateTimestamp, wasExecuted

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Details

LOG

private static final org.slf4j.Logger LOG

metricsSnapshot

private final MetricsSnapshot metricsSnapshot

conf

private org.apache.hadoop.conf.Configuration conf

snapshot

private org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription snapshot

rootDir

private org.apache.hadoop.fs.Path rootDir

snapshotDir

private org.apache.hadoop.fs.Path snapshotDir

workingDir

private org.apache.hadoop.fs.Path workingDir

workingDirFS

private org.apache.hadoop.fs.FileSystem workingDirFS

rootFs

private org.apache.hadoop.fs.FileSystem rootFs

snapshotTable

private TableName snapshotTable

status

private MonitoredTask status

snapshotManifest

private SnapshotManifest snapshotManifest

htd

private TableDescriptor htd

retryCounter

private RetryCounter retryCounter

Constructor Details

SnapshotProcedure

public SnapshotProcedure()

SnapshotProcedure

public SnapshotProcedure(MasterProcedureEnv env,
 org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription snapshot)

Method Details

getTableName

public TableName getTableName()

Description copied from interface: TableProcedureInterface

Returns the name of the table the procedure is operating on

Specified by:

getTableName in interface TableProcedureInterface

Specified by:

getTableName in class AbstractStateMachineTableProcedure<org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProcedureProtos.SnapshotState>

getTableOperationType

public TableProcedureInterface.TableOperationType getTableOperationType()

Description copied from interface: TableProcedureInterface

Given an operation type we can take decisions about what to do with pending operations. e.g. if we get a delete and we have some table operation pending (e.g. add column) we can abort those operations.

Specified by:

getTableOperationType in interface TableProcedureInterface

Specified by:

getTableOperationType in class AbstractStateMachineTableProcedure<org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProcedureProtos.SnapshotState>

Returns:

the operation type that the procedure is executing.

acquireLock

protected Procedure.LockState acquireLock(MasterProcedureEnv env)

Description copied from class: Procedure

The user should override this method if they need a lock on an Entity. A lock can be anything, and it is up to the implementor. The Procedure Framework will call this method just before it invokes Procedure.execute(Object). It calls Procedure.releaseLock(Object) after the call to execute.

If you need to hold the lock for the life of the Procedure -- i.e. you do not want any other Procedure interfering while this Procedure is running, see Procedure.holdLock(Object).

Example: in our Master we can execute request in parallel for different tables. We can create t1 and create t2 and these creates can be executed at the same time. Anything else on t1/t2 is queued waiting that specific table create to happen.

There are 3 LockState:

• LOCK_ACQUIRED should be returned when the proc has the lock and the proc is ready to execute.
• LOCK_YIELD_WAIT should be returned when the proc has not the lock and the framework should take care of readding the procedure back to the runnable set for retry
• LOCK_EVENT_WAIT should be returned when the proc has not the lock and someone will take care of readding the procedure back to the runnable set when the lock is available.

Overrides:

acquireLock in class AbstractStateMachineTableProcedure<org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProcedureProtos.SnapshotState>

Returns:

the lock state as described above.

releaseLock

protected void releaseLock(MasterProcedureEnv env)

Description copied from class: Procedure

The user should override this method, and release lock if necessary.

Overrides:

releaseLock in class AbstractStateMachineTableProcedure<org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProcedureProtos.SnapshotState>

holdLock

protected boolean holdLock(MasterProcedureEnv env)

Description copied from class: Procedure

Used to keep the procedure lock even when the procedure is yielding or suspended.

Overrides:

holdLock in class Procedure<MasterProcedureEnv>

Returns:

true if the procedure should hold on the lock until completionCleanup()

executeFromState

protected StateMachineProcedure.Flow executeFromState(MasterProcedureEnv env,
 org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProcedureProtos.SnapshotState state)
                                               throws ProcedureSuspendedException,
ProcedureYieldException,
InterruptedException

Description copied from class: StateMachineProcedure

called to perform a single step of the specified 'state' of the procedure

Specified by:

executeFromState in class StateMachineProcedure<MasterProcedureEnv,org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProcedureProtos.SnapshotState>

state - state to execute

Returns:

Flow.NO_MORE_STATE if the procedure is completed, Flow.HAS_MORE_STATE if there is another step.

Throws:

ProcedureSuspendedException

ProcedureYieldException

InterruptedException

rollbackState

protected void rollbackState(MasterProcedureEnv env,
 org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProcedureProtos.SnapshotState state)
                      throws IOException,
InterruptedException

Description copied from class: StateMachineProcedure

called to perform the rollback of the specified state

Specified by:

rollbackState in class StateMachineProcedure<MasterProcedureEnv,org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProcedureProtos.SnapshotState>

state - state to rollback

Throws:

IOException - temporary failure, the rollback will retry later

InterruptedException

isRollbackSupported

protected boolean isRollbackSupported(org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProcedureProtos.SnapshotState state)

Description copied from class: StateMachineProcedure

Used by the default implementation of abort() to know if the current state can be aborted and rollback can be triggered.

Overrides:

isRollbackSupported in class StateMachineProcedure<MasterProcedureEnv,org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProcedureProtos.SnapshotState>

getState

protected org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProcedureProtos.SnapshotState getState(int stateId)

Description copied from class: StateMachineProcedure

Convert an ordinal (or state id) to an Enum (or more descriptive) state object.

Specified by:

getState in class StateMachineProcedure<MasterProcedureEnv,org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProcedureProtos.SnapshotState>

Parameters:

stateId - the ordinal() of the state enum (or state id)

Returns:

the state enum object

getStateId

protected int getStateId(org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProcedureProtos.SnapshotState state)

Description copied from class: StateMachineProcedure

Convert the Enum (or more descriptive) state object to an ordinal (or state id).

Specified by:

getStateId in class StateMachineProcedure<MasterProcedureEnv,org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProcedureProtos.SnapshotState>

Parameters:

state - the state enum object

Returns:

stateId the ordinal() of the state enum (or state id)

getInitialState

protected org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProcedureProtos.SnapshotState getInitialState()

Description copied from class: StateMachineProcedure

Return the initial state object that will be used for the first call to executeFromState().

Specified by:

getInitialState in class StateMachineProcedure<MasterProcedureEnv,org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProcedureProtos.SnapshotState>

Returns:

the initial state enum object

prepareSnapshot

private void prepareSnapshot(MasterProcedureEnv env)
                      throws ProcedureSuspendedException,
IOException

Throws:

ProcedureSuspendedException

IOException

prepareSnapshotEnv

private void prepareSnapshotEnv(MasterProcedureEnv env)
                         throws IOException

Throws:

IOException

setTimeoutFailure

protected boolean setTimeoutFailure(MasterProcedureEnv env)

Description copied from class: Procedure

Called by the ProcedureExecutor when the timeout set by setTimeout() is expired.

Another usage for this method is to implement retrying. A procedure can set the state to WAITING_TIMEOUT by calling setState method, and throw a ProcedureSuspendedException to halt the execution of the procedure, and do not forget a call Procedure.setTimeout(int) method to set the timeout. And you should also override this method to wake up the procedure, and also return false to tell the ProcedureExecutor that the timeout event has been handled.

Overrides:

setTimeoutFailure in class Procedure<MasterProcedureEnv>

Returns:

true to let the framework handle the timeout as abort, false in case the procedure handled the timeout itself.

isAnySplitOrMergeProcedureRunning

private boolean isAnySplitOrMergeProcedureRunning(MasterProcedureEnv env)

loadTableDescriptorSnapshot

private TableDescriptor loadTableDescriptorSnapshot(MasterProcedureEnv env)
                                             throws IOException

Throws:

IOException

preSnapshot

private void preSnapshot(MasterProcedureEnv env)
                  throws IOException

Throws:

IOException

postSnapshot

private void postSnapshot(MasterProcedureEnv env)
                   throws IOException

Throws:

IOException

verifySnapshot

private void verifySnapshot(MasterProcedureEnv env)
                     throws IOException

Throws:

IOException

completeSnapshot

private void completeSnapshot(MasterProcedureEnv env)
                       throws IOException

Throws:

IOException

snapshotSplitRegions

private void snapshotSplitRegions(MasterProcedureEnv env)
                           throws IOException

Throws:

IOException

snapshotClosedRegions

private void snapshotClosedRegions(MasterProcedureEnv env)
                            throws IOException

Throws:

IOException

getDefaultRegionReplica

private Stream<RegionInfo> getDefaultRegionReplica(MasterProcedureEnv env)

snapshotSplitOrClosedRegions

private void snapshotSplitOrClosedRegions(MasterProcedureEnv env,
 List<RegionInfo> regions,
 String threadPoolName)
                                   throws IOException

Throws:

IOException

snapshotMobRegion

private void snapshotMobRegion(MasterProcedureEnv env)
                        throws IOException

Throws:

IOException

serializeStateData

protected void serializeStateData(ProcedureStateSerializer serializer)
                           throws IOException

Description copied from class: Procedure

The user-level code of the procedure may have some state to persist (e.g. input arguments or current position in the processing state) to be able to resume on failure.

Overrides:

serializeStateData in class StateMachineProcedure<MasterProcedureEnv,org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProcedureProtos.SnapshotState>

Parameters:

serializer - stores the serializable state

Throws:

IOException

deserializeStateData

protected void deserializeStateData(ProcedureStateSerializer serializer)
                             throws IOException

Description copied from class: Procedure

Called on store load to allow the user to decode the previously serialized state.

Overrides:

deserializeStateData in class StateMachineProcedure<MasterProcedureEnv,org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProcedureProtos.SnapshotState>

Parameters:

serializer - contains the serialized state

Throws:

IOException

createRemoteSnapshotProcedures

private Procedure<MasterProcedureEnv>[] createRemoteSnapshotProcedures(MasterProcedureEnv env)

toStringClassDetails

public void toStringClassDetails(StringBuilder builder)

Description copied from class: Procedure

Extend the toString() information with the procedure details e.g. className and parameters

Overrides:

toStringClassDetails in class AbstractStateMachineTableProcedure<org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProcedureProtos.SnapshotState>

Parameters:

builder - the string builder to use to append the proc specific information

getSnapshotDesc

public org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription getSnapshotDesc()

afterReplay

protected void afterReplay(MasterProcedureEnv env)

Description copied from class: Procedure

Called when the procedure is ready to be added to the queue after the loading/replay operation.

Overrides:

afterReplay in class Procedure<MasterProcedureEnv>

getSnapshot

public org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription getSnapshot()

markSnapshotCorrupted

public void markSnapshotCorrupted()
                           throws IOException

Throws:

IOException

isSnapshotCorrupted

public boolean isSnapshotCorrupted()
                            throws IOException

Throws:

IOException