ServerCrashProcedure (Apache HBase 1.2.12 API)

java.lang.Object
- org.apache.hadoop.hbase.procedure2.Procedure<TEnvironment>
- - org.apache.hadoop.hbase.procedure2.StateMachineProcedure<MasterProcedureEnv,org.apache.hadoop.hbase.protobuf.generated.MasterProcedureProtos.ServerCrashState>
  - - org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure

All Implemented Interfaces:

Comparable<Procedure>, ServerProcedureInterface
```
public class ServerCrashProcedure
extends StateMachineProcedure<MasterProcedureEnv,org.apache.hadoop.hbase.protobuf.generated.MasterProcedureProtos.ServerCrashState>
implements ServerProcedureInterface
```
Handle crashed server. This is a port to ProcedureV2 of what used to be euphemistically called ServerShutdownHandler.
The procedure flow varies dependent on whether meta is assigned, if we are doing distributed log replay versus distributed log splitting, and if we are to split logs at all.
This procedure asks that all crashed servers get processed equally; we yield after the completion of each successful flow step. We do this so that we do not 'deadlock' waiting on a region assignment so we can replay edits which could happen if a region moved there are edits on two servers for replay.
TODO: ASSIGN and WAIT_ON_ASSIGN (at least) are not idempotent. Revisit when assign is pv2. TODO: We do not have special handling for system tables.

Nested Class Summary
- Nested classes/interfaces inherited from class org.apache.hadoop.hbase.procedure2.StateMachineProcedure
  StateMachineProcedure.Flow
- Nested classes/interfaces inherited from interface org.apache.hadoop.hbase.master.procedure.ServerProcedureInterface
  ServerProcedureInterface.ServerOperationType

Field Summary

Fields
Modifier and Type	Field and Description
`private boolean`	`carryingMeta`
`private int`	`cycles` Cycles on same state.
`static int`	`DEFAULT_RETRIES_ON_META`
`static int`	`DEFAULT_SHORT_WAIT_ON_META`
`static int`	`DEFAULT_WAIT_ON_RIT`
`private boolean`	`distributedLogReplay`
`static String`	`KEY_RETRIES_ON_META` Configuration key to set how many retries to cycle before we give up on meta.
`static String`	`KEY_SHORT_WAIT_ON_META` Configuration key to set how long to wait in ms doing a quick check on meta state.
`static String`	`KEY_WAIT_ON_RIT` Configuration key to set how long to wait in ms on regions in transition.
`private static org.apache.commons.logging.Log`	`LOG`
`private static Set<HRegionInfo>`	`META_REGION_SET`
`private boolean`	`notifiedDeadServer` Whether DeadServer knows that we are processing it.
`private int`	`previousState` Ordinal of the previous state.
`private List<HRegionInfo>`	`regionsAssigned` Regions assigned.
`private Set<HRegionInfo>`	`regionsOnCrashedServer` Regions that were on the crashed server.
`private ServerName`	`serverName` Name of the crashed server to process.
`private boolean`	`shouldSplitWal`

Constructor Summary

Constructors
Constructor and Description
`ServerCrashProcedure()` Used when deserializing from a procedure store; we'll construct one of these then call `deserializeStateData(InputStream)`.
`ServerCrashProcedure(MasterProcedureEnv env, ServerName serverName, boolean shouldSplitWal, boolean carryingMeta)` Call this constructor queuing up a Procedure.

Method Summary

Methods
Modifier and Type	Method and Description
`protected boolean`	`abort(MasterProcedureEnv env)` The abort() call is asynchronous and each procedure must decide how to deal with that, if they want to be abortable.
`protected boolean`	`acquireLock(MasterProcedureEnv env)` The user should override this method, and try to take a lock if necessary.
`private boolean`	`assign(MasterProcedureEnv env, List<HRegionInfo> hris)`
`private List<HRegionInfo>`	`calcRegionsToAssign(MasterProcedureEnv env)` Figure out what we need to assign.
`void`	`deserializeStateData(InputStream stream)` Called on store load to allow the user to decode the previously serialized state.
`protected StateMachineProcedure.Flow`	`executeFromState(MasterProcedureEnv env, org.apache.hadoop.hbase.protobuf.generated.MasterProcedureProtos.ServerCrashState state)` called to perform a single step of the specified 'state' of the procedure
`protected org.apache.hadoop.hbase.protobuf.generated.MasterProcedureProtos.ServerCrashState`	`getInitialState()` Return the initial state object that will be used for the first call to executeFromState().
`ServerName`	`getServerName()`
`ServerProcedureInterface.ServerOperationType`	`getServerOperationType()` Given an operation type we can take decisions about what to do with pending operations.
`protected org.apache.hadoop.hbase.protobuf.generated.MasterProcedureProtos.ServerCrashState`	`getState(int stateId)` Convert an ordinal (or state id) to an Enum (or more descriptive) state object.
`protected int`	`getStateId(org.apache.hadoop.hbase.protobuf.generated.MasterProcedureProtos.ServerCrashState state)` Convert the Enum (or more descriptive) state object to an ordinal (or state id).
`boolean`	`hasMetaTableRegion()`
`private boolean`	`isMetaAssignedQuickTest(MasterProcedureEnv env)` A quick test that hbase:meta is assigned; blocks for short time only.
`protected boolean`	`isYieldBeforeExecuteFromState(MasterProcedureEnv env, org.apache.hadoop.hbase.protobuf.generated.MasterProcedureProtos.ServerCrashState state)` For this procedure, yield at end of each successful flow step so that all crashed servers can make progress rather than do the default which has each procedure running to completion before we move to the next.
`private void`	`prepareLogReplay(MasterProcedureEnv env, Set<HRegionInfo> regions)`
`private static boolean`	`processDeadRegion(HRegionInfo hri, AssignmentManager assignmentManager)` Process a dead region from a dead RS.
`private boolean`	`processMeta(MasterProcedureEnv env)`
`protected void`	`releaseLock(MasterProcedureEnv env)` The user should override this method, and release lock if necessary.
`protected void`	`rollbackState(MasterProcedureEnv env, org.apache.hadoop.hbase.protobuf.generated.MasterProcedureProtos.ServerCrashState state)` called to perform the rollback of the specified state
`void`	`serializeStateData(OutputStream stream)` The user-level code of the procedure may have some state to persist (e.g.
`protected boolean`	`shouldWaitClientAck(MasterProcedureEnv env)` By default, the executor will keep the procedure result around util the eviction TTL is expired.
`(package private) static int`	`size(Collection<HRegionInfo> hris)`
`private void`	`splitLogs(MasterProcedureEnv env)`
`private void`	`start(MasterProcedureEnv env)` Start processing of crashed server.
`private void`	`throwProcedureYieldException(String msg)`
`void`	`toStringClassDetails(StringBuilder sb)` Extend the toString() information with the procedure details e.g.
`private void`	`verifyAndAssignMeta(MasterProcedureEnv env)` If hbase:meta is not assigned already, assign.
`private void`	`verifyAndAssignMetaWithRetries(MasterProcedureEnv env)` If hbase:meta is not assigned already, assign.
`private boolean`	`waitOnAssign(MasterProcedureEnv env, List<HRegionInfo> hris)`
`private boolean`	`waitOnRegionToClearRegionsInTransition(AssignmentManager am, HRegionInfo hri, int timeout)`

Methods inherited from class org.apache.hadoop.hbase.procedure2.StateMachineProcedure
execute, isYieldAfterExecutionStep, rollback, setNextState, toStringState

Methods inherited from class org.apache.hadoop.hbase.procedure2.Procedure
addStackIndex, beforeReplay, childrenCountDown, compareTo, completionCleanup, convert, convert, createProcedureInfo, doExecute, doRollback, elapsedTime, getException, getLastUpdate, getNonceKey, getOwner, getParentProcId, getProcId, getProcIdHashCode, getResult, getRootProcedureId, getStackIndexes, getStartTime, getState, getTimeout, getTimeRemaining, hasException, hasOwner, hasParent, hasTimeout, incChildrenLatch, isFailed, isFinished, isSuccess, isWaiting, newInstance, removeStackIndex, setAbortFailure, setChildrenLatch, setFailure, setFailure, setNonceKey, setOwner, setParentProcId, setProcId, setResult, setStackIndexes, setStartTime, setState, setTimeout, setTimeoutFailure, toString, toStringClass, toStringDetails, toStringSimpleSB, updateTimestamp, validateClass, wasExecuted

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

- Field Detail
  - LOG
```
private static final org.apache.commons.logging.Log LOG
```
  - KEY_SHORT_WAIT_ON_META
```
public static final String KEY_SHORT_WAIT_ON_META
```
    Configuration key to set how long to wait in ms doing a quick check on meta state.
    
    See Also:
    Constant Field Values
  - DEFAULT_SHORT_WAIT_ON_META
```
public static final int DEFAULT_SHORT_WAIT_ON_META
```
    See Also:
    Constant Field Values
  - KEY_RETRIES_ON_META
```
public static final String KEY_RETRIES_ON_META
```
    Configuration key to set how many retries to cycle before we give up on meta. Each attempt will wait at least KEY_SHORT_WAIT_ON_META milliseconds.
    
    See Also:
    Constant Field Values
  - DEFAULT_RETRIES_ON_META
```
public static final int DEFAULT_RETRIES_ON_META
```
    See Also:
    Constant Field Values
  - KEY_WAIT_ON_RIT
```
public static final String KEY_WAIT_ON_RIT
```
    Configuration key to set how long to wait in ms on regions in transition.
    
    See Also:
    Constant Field Values
  - DEFAULT_WAIT_ON_RIT
```
public static final int DEFAULT_WAIT_ON_RIT
```
    See Also:
    Constant Field Values
  - META_REGION_SET
```
private static final Set<HRegionInfo> META_REGION_SET
```
  - serverName
```
private ServerName serverName
```
    Name of the crashed server to process.
  - notifiedDeadServer
```
private boolean notifiedDeadServer
```
    Whether DeadServer knows that we are processing it.
  - regionsOnCrashedServer
```
private Set<HRegionInfo> regionsOnCrashedServer
```
    Regions that were on the crashed server.
  - regionsAssigned
```
private List<HRegionInfo> regionsAssigned
```
    Regions assigned. Usually some subset of regionsOnCrashedServer.
  - distributedLogReplay
```
private boolean distributedLogReplay
```
  - carryingMeta
```
private boolean carryingMeta
```
  - shouldSplitWal
```
private boolean shouldSplitWal
```
  - cycles
```
private int cycles
```
    Cycles on same state. Good for figuring if we are stuck.
  - previousState
```
private int previousState
```
    Ordinal of the previous state. So we can tell if we are progressing or not. TODO: if useful, move this back up into StateMachineProcedure
- Constructor Detail
  - ServerCrashProcedure
```
public ServerCrashProcedure(MasterProcedureEnv env,
                    ServerName serverName,
                    boolean shouldSplitWal,
                    boolean carryingMeta)
```
    Call this constructor queuing up a Procedure.
    
    Parameters:
    serverName - Name of the crashed server.
    shouldSplitWal - True if we should split WALs as part of crashed server processing.
    carryingMeta - True if carrying hbase:meta table region.
  - ServerCrashProcedure
```
public ServerCrashProcedure()
```
    Used when deserializing from a procedure store; we'll construct one of these then call deserializeStateData(InputStream). Do not use directly.
- Method Detail
  - throwProcedureYieldException
```
private void throwProcedureYieldException(String msg)
                                   throws ProcedureYieldException
```
    Throws:
    
    ProcedureYieldException
  - executeFromState
```
protected StateMachineProcedure.Flow executeFromState(MasterProcedureEnv env,
                                          org.apache.hadoop.hbase.protobuf.generated.MasterProcedureProtos.ServerCrashState state)
                                               throws ProcedureYieldException
```
    Description copied from class: StateMachineProcedure
    
    called to perform a single step of the specified 'state' of the procedure
    
    Specified by:
    
    executeFromState in class StateMachineProcedure<MasterProcedureEnv,org.apache.hadoop.hbase.protobuf.generated.MasterProcedureProtos.ServerCrashState>
    
    state - state to execute
    
    Returns:
    Flow.NO_MORE_STATE if the procedure is completed, Flow.HAS_MORE_STATE if there is another step.
    
    Throws:
    
    ProcedureYieldException
  - start
```
private void start(MasterProcedureEnv env)
            throws IOException
```
    Start processing of crashed server. In here we'll just set configs. and return.
    
    Parameters:
    env -
    
    Throws:
    
    IOException
  - processMeta
```
private boolean processMeta(MasterProcedureEnv env)
                     throws IOException
```
    Parameters:
    env -
    
    Returns:
    False if we fail to assign and split logs on meta ('process').
    
    Throws:
    
    IOException
    
    InterruptedException
  - waitOnRegionToClearRegionsInTransition
```
private boolean waitOnRegionToClearRegionsInTransition(AssignmentManager am,
                                             HRegionInfo hri,
                                             int timeout)
                                                throws InterruptedIOException
```
    Returns:
    True if region cleared RIT, else false if we timed out waiting.
    
    Throws:
    
    InterruptedIOException
  - prepareLogReplay
```
private void prepareLogReplay(MasterProcedureEnv env,
                    Set<HRegionInfo> regions)
                       throws IOException
```
    Throws:
    
    IOException
  - splitLogs
```
private void splitLogs(MasterProcedureEnv env)
                throws IOException
```
    Throws:
    
    IOException
  - size
```
static int size(Collection<HRegionInfo> hris)
```
  - calcRegionsToAssign
```
private List<HRegionInfo> calcRegionsToAssign(MasterProcedureEnv env)
                                       throws IOException
```
    Figure out what we need to assign. Should be idempotent.
    
    Parameters:
    env -
    
    Returns:
    List of calculated regions to assign; may be empty or null.
    
    Throws:
    
    IOException
  - assign
```
private boolean assign(MasterProcedureEnv env,
             List<HRegionInfo> hris)
                throws InterruptedIOException
```
    Throws:
    
    InterruptedIOException
  - waitOnAssign
```
private boolean waitOnAssign(MasterProcedureEnv env,
                   List<HRegionInfo> hris)
                      throws InterruptedIOException
```
    Throws:
    
    InterruptedIOException
  - rollbackState
```
protected void rollbackState(MasterProcedureEnv env,
                 org.apache.hadoop.hbase.protobuf.generated.MasterProcedureProtos.ServerCrashState state)
                      throws IOException
```
    Description copied from class: StateMachineProcedure
    
    called to perform the rollback of the specified state
    
    Specified by:
    
    rollbackState in class StateMachineProcedure<MasterProcedureEnv,org.apache.hadoop.hbase.protobuf.generated.MasterProcedureProtos.ServerCrashState>
    
    state - state to rollback
    
    Throws:
    
    IOException - temporary failure, the rollback will retry later
  - getState
```
protected org.apache.hadoop.hbase.protobuf.generated.MasterProcedureProtos.ServerCrashState getState(int stateId)
```
    Description copied from class: StateMachineProcedure
    
    Convert an ordinal (or state id) to an Enum (or more descriptive) state object.
    
    Specified by:
    
    getState in class StateMachineProcedure<MasterProcedureEnv,org.apache.hadoop.hbase.protobuf.generated.MasterProcedureProtos.ServerCrashState>
    
    Parameters:
    stateId - the ordinal() of the state enum (or state id)
    
    Returns:
    the state enum object
  - getStateId
```
protected int getStateId(org.apache.hadoop.hbase.protobuf.generated.MasterProcedureProtos.ServerCrashState state)
```
    Description copied from class: StateMachineProcedure
    
    Convert the Enum (or more descriptive) state object to an ordinal (or state id).
    
    Specified by:
    
    getStateId in class StateMachineProcedure<MasterProcedureEnv,org.apache.hadoop.hbase.protobuf.generated.MasterProcedureProtos.ServerCrashState>
    
    Parameters:
    state - the state enum object
    
    Returns:
    stateId the ordinal() of the state enum (or state id)
  - getInitialState
```
protected org.apache.hadoop.hbase.protobuf.generated.MasterProcedureProtos.ServerCrashState getInitialState()
```
    Description copied from class: StateMachineProcedure
    
    Return the initial state object that will be used for the first call to executeFromState().
    
    Specified by:
    
    getInitialState in class StateMachineProcedure<MasterProcedureEnv,org.apache.hadoop.hbase.protobuf.generated.MasterProcedureProtos.ServerCrashState>
    
    Returns:
    the initial state enum object
  - abort
```
protected boolean abort(MasterProcedureEnv env)
```
    Description copied from class: Procedure
    
    The abort() call is asynchronous and each procedure must decide how to deal with that, if they want to be abortable. The simplest implementation is to have an AtomicBoolean set in the abort() method and then the execute() will check if the abort flag is set or not. abort() may be called multiple times from the client, so the implementation must be idempotent. NOTE: abort() is not like Thread.interrupt() it is just a notification that allows the procedure implementor where to abort to avoid leak and have a better control on what was executed and what not.
    
    Specified by:
    
    abort in class Procedure<MasterProcedureEnv>
  - acquireLock
```
protected boolean acquireLock(MasterProcedureEnv env)
```
    Description copied from class: Procedure
    
    The user should override this method, and try to take a lock if necessary. A lock can be anything, and it is up to the implementor. Example: in our Master we can execute request in parallel for different tables create t1 and create t2 can be executed at the same time. anything else on t1/t2 is queued waiting that specific table create to happen.
    
    Overrides:
    
    acquireLock in class Procedure<MasterProcedureEnv>
    
    Returns:
    true if the lock was acquired and false otherwise
  - releaseLock
```
protected void releaseLock(MasterProcedureEnv env)
```
    Description copied from class: Procedure
    
    The user should override this method, and release lock if necessary.
    
    Overrides:
    
    releaseLock in class Procedure<MasterProcedureEnv>
  - toStringClassDetails
```
public void toStringClassDetails(StringBuilder sb)
```
    Description copied from class: Procedure
    
    Extend the toString() information with the procedure details e.g. className and parameters
    
    Overrides:
    
    toStringClassDetails in class Procedure<MasterProcedureEnv>
    
    Parameters:
    sb - the string builder to use to append the proc specific information
  - serializeStateData
```
public void serializeStateData(OutputStream stream)
                        throws IOException
```
    Description copied from class: Procedure
    
    The user-level code of the procedure may have some state to persist (e.g. input arguments) to be able to resume on failure.
    
    Overrides:
    
    serializeStateData in class StateMachineProcedure<MasterProcedureEnv,org.apache.hadoop.hbase.protobuf.generated.MasterProcedureProtos.ServerCrashState>
    
    Parameters:
    stream - the stream that will contain the user serialized data
    
    Throws:
    
    IOException
  - deserializeStateData
```
public void deserializeStateData(InputStream stream)
                          throws IOException
```
    Description copied from class: Procedure
    
    Called on store load to allow the user to decode the previously serialized state.
    
    Overrides:
    
    deserializeStateData in class StateMachineProcedure<MasterProcedureEnv,org.apache.hadoop.hbase.protobuf.generated.MasterProcedureProtos.ServerCrashState>
    
    Parameters:
    stream - the stream that contains the user serialized data
    
    Throws:
    
    IOException
  - processDeadRegion
```
private static boolean processDeadRegion(HRegionInfo hri,
                        AssignmentManager assignmentManager)
                                  throws IOException
```
    Process a dead region from a dead RS. Checks if the region is disabled or disabling or if the region has a partially completed split.
    
    Parameters:
    hri -
    assignmentManager -
    
    Returns:
    Returns true if specified region should be assigned, false if not.
    
    Throws:
    
    IOException
  - verifyAndAssignMetaWithRetries
```
private void verifyAndAssignMetaWithRetries(MasterProcedureEnv env)
                                     throws IOException
```
    If hbase:meta is not assigned already, assign.
    
    Throws:
    
    IOException
  - verifyAndAssignMeta
```
private void verifyAndAssignMeta(MasterProcedureEnv env)
                          throws InterruptedException,
                                 IOException,
                                 org.apache.zookeeper.KeeperException
```
    If hbase:meta is not assigned already, assign.
    
    Throws:
    
    InterruptedException
    
    IOException
    
    org.apache.zookeeper.KeeperException
  - isMetaAssignedQuickTest
```
private boolean isMetaAssignedQuickTest(MasterProcedureEnv env)
                                 throws InterruptedException,
                                        IOException
```
    A quick test that hbase:meta is assigned; blocks for short time only.
    
    Returns:
    True if hbase:meta location is available and verified as good.
    
    Throws:
    
    InterruptedException
    
    IOException
  - getServerName
```
public ServerName getServerName()
```
    Specified by:
    
    getServerName in interface ServerProcedureInterface
    
    Returns:
    Name of this server instance.
  - hasMetaTableRegion
```
public boolean hasMetaTableRegion()
```
    Specified by:
    
    hasMetaTableRegion in interface ServerProcedureInterface
    
    Returns:
    True if this server has an hbase:meta table region.
  - getServerOperationType
```
public ServerProcedureInterface.ServerOperationType getServerOperationType()
```
    Description copied from interface: ServerProcedureInterface
    
    Given an operation type we can take decisions about what to do with pending operations. e.g. if we get a crash handler and we have some assignment operation pending we can abort those operations.
    
    Specified by:
    
    getServerOperationType in interface ServerProcedureInterface
    
    Returns:
    the operation type that the procedure is executing.
  - isYieldBeforeExecuteFromState
```
protected boolean isYieldBeforeExecuteFromState(MasterProcedureEnv env,
                                    org.apache.hadoop.hbase.protobuf.generated.MasterProcedureProtos.ServerCrashState state)
```
    For this procedure, yield at end of each successful flow step so that all crashed servers can make progress rather than do the default which has each procedure running to completion before we move to the next. For crashed servers, especially if running with distributed log replay, we will want all servers to come along; we do not want the scenario where a server is stuck waiting for regions to online so it can replay edits.
    
    Overrides:
    
    isYieldBeforeExecuteFromState in class StateMachineProcedure<MasterProcedureEnv,org.apache.hadoop.hbase.protobuf.generated.MasterProcedureProtos.ServerCrashState>
    
    state - the state we are going to execute next.
    
    Returns:
    Return true if the executor should yield before the execution of the specified step. Defaults to return false.
  - shouldWaitClientAck
```
protected boolean shouldWaitClientAck(MasterProcedureEnv env)
```
    Description copied from class: Procedure
    
    By default, the executor will keep the procedure result around util the eviction TTL is expired. The client can cut down the waiting time by requesting that the result is removed from the executor. In case of system started procedure, we can force the executor to auto-ack.
    
    Overrides:
    
    shouldWaitClientAck in class Procedure<MasterProcedureEnv>
    
    Parameters:
    env - the environment passed to the ProcedureExecutor
    
    Returns:
    true if the executor should wait the client ack for the result. Defaults to return true.

Class ServerCrashProcedure

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.hbase.procedure2.StateMachineProcedure

Nested classes/interfaces inherited from interface org.apache.hadoop.hbase.master.procedure.ServerProcedureInterface

Field Summary

Constructor Summary

Method Summary

Methods inherited from class org.apache.hadoop.hbase.procedure2.StateMachineProcedure

Methods inherited from class org.apache.hadoop.hbase.procedure2.Procedure

Methods inherited from class java.lang.Object

Field Detail

LOG

KEY_SHORT_WAIT_ON_META

DEFAULT_SHORT_WAIT_ON_META

KEY_RETRIES_ON_META

DEFAULT_RETRIES_ON_META

KEY_WAIT_ON_RIT

DEFAULT_WAIT_ON_RIT

META_REGION_SET

serverName

notifiedDeadServer

regionsOnCrashedServer

regionsAssigned

distributedLogReplay

carryingMeta

shouldSplitWal

cycles

previousState

Constructor Detail

ServerCrashProcedure

ServerCrashProcedure

Method Detail

throwProcedureYieldException

executeFromState

start

processMeta

waitOnRegionToClearRegionsInTransition

prepareLogReplay

splitLogs

size

calcRegionsToAssign

assign

waitOnAssign

rollbackState

getState

getStateId

getInitialState

abort

acquireLock

releaseLock

toStringClassDetails

serializeStateData

deserializeStateData

processDeadRegion

verifyAndAssignMetaWithRetries

verifyAndAssignMeta

isMetaAssignedQuickTest

getServerName

hasMetaTableRegion

getServerOperationType

isYieldBeforeExecuteFromState

shouldWaitClientAck