org.apache.hadoop.hbase.client

Class PreemptiveFastFailInterceptor

java.lang.Object

org.apache.hadoop.hbase.client.RetryingCallerInterceptor

org.apache.hadoop.hbase.client.PreemptiveFastFailInterceptor

@InterfaceAudience.Private
class PreemptiveFastFailInterceptor
extends RetryingCallerInterceptor

The concrete RetryingCallerInterceptor class that implements the preemptive fast fail feature.

The motivation is as follows : In case where a large number of clients try and talk to a particular region server in hbase, if the region server goes down due to network problems, we might end up in a scenario where the clients would go into a state where they all start to retry. This behavior will set off many of the threads in pretty much the same path and they all would be sleeping giving rise to a state where the client either needs to create more threads to send new requests to other hbase machines or block because the client cannot create anymore threads.

In most cases the clients might prefer to have a bound on the number of threads that are created in order to send requests to hbase. This would mostly result in the client thread starvation.

To circumvent this problem, the approach that is being taken here under is to let 1 of the many threads who are trying to contact the regionserver with connection problems and let the other threads get a PreemptiveFastFailException so that they can move on and take other requests.

This would give the client more flexibility on the kind of action he would want to take in cases where the regionserver is down. He can either discard the requests and send a nack upstream faster or have an application level retry or buffer the requests up so as to send them down to hbase later.

Field Summary

Fields

Modifier and Type	Field and Description
protected long	failureMapCleanupIntervalMilliSec
private long	fastFailClearingTimeMilliSec
protected long	fastFailThresholdMilliSec
protected long	lastFailureMapCleanupTimeMilliSec
private static org.slf4j.Logger	LOG
protected ConcurrentMap<ServerName,FailureInfo>	repeatedFailuresMap
private ThreadLocal<org.apache.commons.lang3.mutable.MutableBoolean>	threadRetryingInFastFailMode

Constructor Summary

Constructors

Constructor and Description
PreemptiveFastFailInterceptor(org.apache.hadoop.conf.Configuration conf)

Method Summary

Modifier and Type	Method and Description
RetryingCallerInterceptorContext	createEmptyContext() This returns the context object for the current call.
private boolean	currentThreadInFastFailMode() Checks to see if the current thread is already in FastFail mode for *some* server.
void	handleFailure(FastFailInterceptorContext context, Throwable t)
void	handleFailure(RetryingCallerInterceptorContext context, Throwable t) Call this function in case we caught a failure during retries.
protected void	handleFailureToServer(ServerName serverName, Throwable t) Handles failures encountered when communicating with a server.
void	handleThrowable(Throwable t1, ServerName serverName, org.apache.commons.lang3.mutable.MutableBoolean couldNotCommunicateWithServer, org.apache.commons.lang3.mutable.MutableBoolean guaranteedClientSideOnly)
private boolean	inFastFailMode(ServerName server) Checks to see if we are in the Fast fail mode for requests to the server.
void	intercept(FastFailInterceptorContext context)
void	intercept(RetryingCallerInterceptorContext context) Call this function alongside the actual call done on the callable.
protected boolean	isServerInFailureMap(ServerName serverName)
protected void	occasionallyCleanupFailureInformation() Occasionally cleans up unused information in repeatedFailuresMap.
protected boolean	shouldRetryInspiteOfFastFail(FailureInfo fInfo) Check to see if the client should try to connnect to the server, inspite of knowing that it is in the fast fail mode.
String	toString()
void	updateFailureInfo(FastFailInterceptorContext context)
void	updateFailureInfo(RetryingCallerInterceptorContext context) Call this function to update at the end of the retry.
private void	updateFailureInfoForServer(ServerName server, FailureInfo fInfo, boolean didTry, boolean couldNotCommunicate, boolean retryDespiteFastFailMode) This function updates the Failure info for a particular server after the attempt to

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

LOG

private static final org.slf4j.Logger LOG

fastFailThresholdMilliSec

protected final long fastFailThresholdMilliSec

repeatedFailuresMap

protected final ConcurrentMap<ServerName,FailureInfo> repeatedFailuresMap

failureMapCleanupIntervalMilliSec

protected final long failureMapCleanupIntervalMilliSec

lastFailureMapCleanupTimeMilliSec

protected volatile long lastFailureMapCleanupTimeMilliSec

fastFailClearingTimeMilliSec

private long fastFailClearingTimeMilliSec

threadRetryingInFastFailMode

private final ThreadLocal<org.apache.commons.lang3.mutable.MutableBoolean> threadRetryingInFastFailMode

Constructor Detail

PreemptiveFastFailInterceptor

public PreemptiveFastFailInterceptor(org.apache.hadoop.conf.Configuration conf)

Method Detail

intercept

public void intercept(FastFailInterceptorContext context)
               throws PreemptiveFastFailException

Throws:

PreemptiveFastFailException

handleFailure

public void handleFailure(FastFailInterceptorContext context,
                          Throwable t)
                   throws IOException

Throws:

IOException

updateFailureInfo

public void updateFailureInfo(FastFailInterceptorContext context)

handleFailureToServer

protected void handleFailureToServer(ServerName serverName,
                                     Throwable t)

Handles failures encountered when communicating with a server. Updates the FailureInfo in repeatedFailuresMap to reflect the failure. Throws RepeatedConnectException if the client is in Fast fail mode.

Parameters:

serverName -

t - - the throwable to be handled.

Throws:

PreemptiveFastFailException

handleThrowable

public void handleThrowable(Throwable t1,
                            ServerName serverName,
                            org.apache.commons.lang3.mutable.MutableBoolean couldNotCommunicateWithServer,
                            org.apache.commons.lang3.mutable.MutableBoolean guaranteedClientSideOnly)
                     throws IOException

Throws:

IOException

occasionallyCleanupFailureInformation

protected void occasionallyCleanupFailureInformation()

Occasionally cleans up unused information in repeatedFailuresMap. repeatedFailuresMap stores the failure information for all remote hosts that had failures. In order to avoid these from growing indefinitely, occassionallyCleanupFailureInformation() will clear these up once every cleanupInterval ms.

inFastFailMode

private boolean inFastFailMode(ServerName server)

Checks to see if we are in the Fast fail mode for requests to the server. If a client is unable to contact a server for more than fastFailThresholdMilliSec the client will get into fast fail mode.

Parameters:

server -

Returns:

true if the client is in fast fail mode for the server.

currentThreadInFastFailMode

private boolean currentThreadInFastFailMode()

Checks to see if the current thread is already in FastFail mode for *some* server.

Returns:

true, if the thread is already in FF mode.

shouldRetryInspiteOfFastFail

protected boolean shouldRetryInspiteOfFastFail(FailureInfo fInfo)

Check to see if the client should try to connnect to the server, inspite of knowing that it is in the fast fail mode. The idea here is that we want just one client thread to be actively trying to reconnect, while all the other threads trying to reach the server will short circuit.

Parameters:

fInfo -

Returns:

true if the client should try to connect to the server.

updateFailureInfoForServer

private void updateFailureInfoForServer(ServerName server,
                                        FailureInfo fInfo,
                                        boolean didTry,
                                        boolean couldNotCommunicate,
                                        boolean retryDespiteFastFailMode)

This function updates the Failure info for a particular server after the attempt to

Parameters:

server -

fInfo -

couldNotCommunicate -

retryDespiteFastFailMode -

intercept

public void intercept(RetryingCallerInterceptorContext context)
               throws PreemptiveFastFailException

Description copied from class: RetryingCallerInterceptor

Call this function alongside the actual call done on the callable.

Specified by:

intercept in class RetryingCallerInterceptor

Throws:

PreemptiveFastFailException

handleFailure

public void handleFailure(RetryingCallerInterceptorContext context,
                          Throwable t)
                   throws IOException

Description copied from class: RetryingCallerInterceptor

Call this function in case we caught a failure during retries.

Specified by:

handleFailure in class RetryingCallerInterceptor

Parameters:

context - : The context object that we obtained previously.

t - : The exception that we caught in this particular try

Throws:

IOException

updateFailureInfo

public void updateFailureInfo(RetryingCallerInterceptorContext context)

Description copied from class: RetryingCallerInterceptor

Call this function to update at the end of the retry. This is not necessary to happen.

Specified by:

updateFailureInfo in class RetryingCallerInterceptor

createEmptyContext

public RetryingCallerInterceptorContext createEmptyContext()

Description copied from class: RetryingCallerInterceptor

This returns the context object for the current call.

Specified by:

createEmptyContext in class RetryingCallerInterceptor

Returns:

context : the context that needs to be used during this call.

isServerInFailureMap

protected boolean isServerInFailureMap(ServerName serverName)

toString

public String toString()

Specified by:

toString in class RetryingCallerInterceptor