Class ReplicationSourceWALReader
java.lang.Object
java.lang.Thread
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader
- All Implemented Interfaces:
Runnable
- Direct Known Subclasses:
SerialReplicationSourceWALReader
Reads and filters WAL entries, groups the filtered entries into batches, and puts the batches
onto a queue
-
Nested Class Summary
Nested classes/interfaces inherited from class java.lang.Thread
Thread.State, Thread.UncaughtExceptionHandler
-
Field Summary
Modifier and TypeFieldDescriptionprivate final org.apache.hadoop.conf.Configuration
private long
(package private) final BlockingQueue<WALEntryBatch>
private final WALEntryFilter
private final org.apache.hadoop.fs.FileSystem
private boolean
private static final org.slf4j.Logger
private final ReplicationSourceLogQueue
private final int
private final int
private final long
private final long
private final ReplicationSource
private final String
Fields inherited from class java.lang.Thread
MAX_PRIORITY, MIN_PRIORITY, NORM_PRIORITY
-
Constructor Summary
ConstructorDescriptionReplicationSourceWALReader
(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.conf.Configuration conf, ReplicationSourceLogQueue logQueue, long startPosition, WALEntryFilter filter, ReplicationSource source, String walGroupId) Creates a reader worker for a given WAL queue. -
Method Summary
Modifier and TypeMethodDescriptionprotected final boolean
addEntryToBatch
(WALEntryBatch batch, WAL.Entry entry) private boolean
Count the number of different row keys in the given edit because of mini-batching.private WALEntryBatch
createBatch
(WALEntryStream entryStream) protected final WAL.Entry
filterEntry
(WAL.Entry entry) org.apache.hadoop.fs.Path
private long
private ReplicationSourceManager
boolean
Returns whether the reader thread is runningpoll
(long timeout) protected void
readWALEntries
(WALEntryStream entryStream, WALEntryBatch batch) private void
void
run()
void
setReaderRunning
(boolean readerRunning) private int
Calculate the total size of all the store filesprotected final int
sleep
(int sleepMultiplier) protected static final boolean
switched
(WALEntryStream entryStream, org.apache.hadoop.fs.Path path) take()
Retrieves the next batch of WAL entries from the queue, waiting up to the specified time for a batch to become availableprivate void
updateBatchStats
(WALEntryBatch batch, WAL.Entry entry, long entrySize) private void
updateReplicationMarkerEdit
(WAL.Entry entry, long offset) Methods inherited from class java.lang.Thread
activeCount, checkAccess, clone, countStackFrames, currentThread, dumpStack, enumerate, getAllStackTraces, getContextClassLoader, getDefaultUncaughtExceptionHandler, getId, getName, getPriority, getStackTrace, getState, getThreadGroup, getUncaughtExceptionHandler, holdsLock, interrupt, interrupted, isAlive, isDaemon, isInterrupted, join, join, join, onSpinWait, resume, setContextClassLoader, setDaemon, setDefaultUncaughtExceptionHandler, setName, setPriority, setUncaughtExceptionHandler, sleep, sleep, start, stop, suspend, toString, yield
-
Field Details
-
LOG
-
logQueue
-
fs
-
conf
-
filter
-
source
-
entryBatchQueue
-
replicationBatchSizeCapacity
-
replicationBatchCountCapacity
-
currentPosition
-
sleepForRetries
-
maxRetriesMultiplier
-
isReaderRunning
-
walGroupId
-
-
Constructor Details
-
ReplicationSourceWALReader
public ReplicationSourceWALReader(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.conf.Configuration conf, ReplicationSourceLogQueue logQueue, long startPosition, WALEntryFilter filter, ReplicationSource source, String walGroupId) Creates a reader worker for a given WAL queue. Reads WAL entries off a given queue, batches the entries, and puts them on a batch queue.- Parameters:
fs
- the files system to useconf
- configuration to uselogQueue
- The WAL queue to read off ofstartPosition
- position in the first WAL to start reading fromfilter
- The filter to use while readingsource
- replication source
-
-
Method Details
-
replicationDone
- Throws:
InterruptedException
-
sleep
-
run
-
addEntryToBatch
-
switched
-
readWALEntries
protected void readWALEntries(WALEntryStream entryStream, WALEntryBatch batch) throws InterruptedException - Throws:
InterruptedException
-
getCurrentPath
-
checkBufferQuota
-
createBatch
-
filterEntry
-
take
Retrieves the next batch of WAL entries from the queue, waiting up to the specified time for a batch to become available- Returns:
- A batch of entries, along with the position in the log after reading the batch
- Throws:
InterruptedException
- if interrupted while waiting
-
poll
- Throws:
InterruptedException
-
getEntrySizeIncludeBulkLoad
-
updateBatchStats
-
countDistinctRowKeysAndHFiles
Count the number of different row keys in the given edit because of mini-batching. We assume that there's at least one Cell in the WALEdit.- Parameters:
edit
- edit to count row keys from- Returns:
- number of different row keys and HFiles
-
sizeOfStoreFilesIncludeBulkLoad
Calculate the total size of all the store files- Parameters:
edit
- edit to count row keys from- Returns:
- the total size of the store files
-
updateReplicationMarkerEdit
-
isReaderRunning
Returns whether the reader thread is running -
setReaderRunning
- Parameters:
readerRunning
- the readerRunning to set
-
getSourceManager
-