org.apache.hadoop.hbase.replication.regionserver

Class ReplicationSourceShipper

java.lang.Object

java.lang.Thread

org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper

All Implemented Interfaces:

Runnable

Direct Known Subclasses:

RecoveredReplicationSourceShipper

@InterfaceAudience.Private
public class ReplicationSourceShipper
extends Thread

This thread reads entries from a queue and ships them. Entries are placed onto the queue by ReplicationSourceWALReaderThread

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	ReplicationSourceShipper.WorkerState

Nested classes/interfaces inherited from class java.lang.Thread

Thread.State, Thread.UncaughtExceptionHandler

Field Summary

Fields

Modifier and Type	Field and Description
private org.apache.hadoop.conf.Configuration	conf
private org.apache.hadoop.fs.Path	currentPath
private long	currentPosition
private int	DEFAULT_TIMEOUT
protected ReplicationSourceWALReader	entryReader
private int	getEntriesTimeout
private static org.slf4j.Logger	LOG
protected ReplicationSourceLogQueue	logQueue
protected int	maxRetriesMultiplier
private int	shipEditsTimeout
protected long	sleepForRetries
private ReplicationSource	source
private ReplicationSourceShipper.WorkerState	state
protected String	walGroupId

Fields inherited from class java.lang.Thread

MAX_PRIORITY, MIN_PRIORITY, NORM_PRIORITY

Constructor Summary

Constructors

Constructor and Description
ReplicationSourceShipper(org.apache.hadoop.conf.Configuration conf, String walGroupId, ReplicationSourceLogQueue logQueue, ReplicationSource source)

Method Summary

Modifier and Type	Method and Description
private void	cleanUpHFileRefs(WALEdit edit)
(package private) void	clearWALEntryBatch() Attempts to properly update ReplicationSourceManager.totalBufferUser, in case there were unprocessed entries batched by the reader to the shipper, but the shipper didn't manage to ship those because the replication source is being terminated.
private int	getBatchEntrySizeExcludeBulkLoad(WALEntryBatch entryBatch) get batchEntry size excludes bulk load file sizes.
(package private) org.apache.hadoop.fs.Path	getCurrentPath()
(package private) long	getCurrentPosition()
(package private) long	getStartPosition()
protected boolean	isActive()
boolean	isFinished()
protected void	noMoreData()
protected void	postFinish()
void	run()
(package private) void	setWALReader(ReplicationSourceWALReader entryReader)
protected void	setWorkerState(ReplicationSourceShipper.WorkerState state)
private void	shipEdits(WALEntryBatch entryBatch) Do the shipping logic
boolean	sleepForRetries(String msg, int sleepMultiplier) Do the sleeping logic
void	startup(Thread.UncaughtExceptionHandler handler)
(package private) void	stopWorker()
private boolean	updateLogPosition(WALEntryBatch batch)

Methods inherited from class java.lang.Thread

activeCount, checkAccess, clone, countStackFrames, currentThread, destroy, dumpStack, enumerate, getAllStackTraces, getContextClassLoader, getDefaultUncaughtExceptionHandler, getId, getName, getPriority, getStackTrace, getState, getThreadGroup, getUncaughtExceptionHandler, holdsLock, interrupt, interrupted, isAlive, isDaemon, isInterrupted, join, join, join, resume, setContextClassLoader, setDaemon, setDefaultUncaughtExceptionHandler, setName, setPriority, setUncaughtExceptionHandler, sleep, sleep, start, stop, stop, suspend, toString, yield

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

LOG

private static final org.slf4j.Logger LOG

conf

private final org.apache.hadoop.conf.Configuration conf

walGroupId

protected final String walGroupId

logQueue

protected final ReplicationSourceLogQueue logQueue

source

private final ReplicationSource source

currentPosition

private volatile long currentPosition

currentPath

private org.apache.hadoop.fs.Path currentPath

state

private volatile ReplicationSourceShipper.WorkerState state

entryReader

protected ReplicationSourceWALReader entryReader

sleepForRetries

protected final long sleepForRetries

maxRetriesMultiplier

protected final int maxRetriesMultiplier

DEFAULT_TIMEOUT

private final int DEFAULT_TIMEOUT

See Also:

Constant Field Values

getEntriesTimeout

private final int getEntriesTimeout

shipEditsTimeout

private final int shipEditsTimeout

Constructor Detail

ReplicationSourceShipper

public ReplicationSourceShipper(org.apache.hadoop.conf.Configuration conf,
                                String walGroupId,
                                ReplicationSourceLogQueue logQueue,
                                ReplicationSource source)

Method Detail

run

public final void run()

Specified by:

run in interface Runnable

Overrides:

run in class Thread

noMoreData

protected void noMoreData()

postFinish

protected void postFinish()

getBatchEntrySizeExcludeBulkLoad

private int getBatchEntrySizeExcludeBulkLoad(WALEntryBatch entryBatch)

get batchEntry size excludes bulk load file sizes. Uses ReplicationSourceWALReader's static method.

shipEdits

private void shipEdits(WALEntryBatch entryBatch)

Do the shipping logic

cleanUpHFileRefs

private void cleanUpHFileRefs(WALEdit edit)
                       throws IOException

Throws:

IOException

updateLogPosition

private boolean updateLogPosition(WALEntryBatch batch)

startup

public void startup(Thread.UncaughtExceptionHandler handler)

getCurrentPath

org.apache.hadoop.fs.Path getCurrentPath()

getCurrentPosition

long getCurrentPosition()

setWALReader

void setWALReader(ReplicationSourceWALReader entryReader)

getStartPosition

long getStartPosition()

isActive

protected boolean isActive()

setWorkerState

protected final void setWorkerState(ReplicationSourceShipper.WorkerState state)

stopWorker

void stopWorker()

isFinished

public boolean isFinished()

sleepForRetries

public boolean sleepForRetries(String msg,
                               int sleepMultiplier)

Do the sleeping logic

Parameters:

msg - Why we sleep

sleepMultiplier - by how many times the default sleeping time is augmented

Returns:

True if sleepMultiplier is < maxRetriesMultiplier

clearWALEntryBatch

void clearWALEntryBatch()

Attempts to properly update ReplicationSourceManager.totalBufferUser, in case there were unprocessed entries batched by the reader to the shipper, but the shipper didn't manage to ship those because the replication source is being terminated. In that case, it iterates through the batched entries and decrease the pending entries size from ReplicationSourceManager.totalBufferUser

NOTES 1) This method should only be called upon replication source termination. It blocks waiting for both shipper and reader threads termination, to make sure no race conditions when updating ReplicationSourceManager.totalBufferUser. 2) It does not attempt to terminate reader and shipper threads. Those must have been triggered interruption/termination prior to calling this method.