org.apache.hadoop.hbase.wal

Class WALSplitter.OutputSink

java.lang.Object

org.apache.hadoop.hbase.wal.WALSplitter.OutputSink

Direct Known Subclasses:

RegionReplicaReplicationEndpoint.RegionReplicaOutputSink, WALSplitter.LogRecoveredEditsOutputSink

Enclosing class:

WALSplitter

public abstract static class WALSplitter.OutputSink
extends Object

The following class is an abstraction class to provide a common interface to support different ways of consuming recovered edits.

Field Summary

Fields

Modifier and Type	Field and Description
protected Set<byte[]>	blacklistedRegions
protected boolean	closeAndCleanCompleted
protected WALSplitter.PipelineController	controller
protected WALSplitter.EntryBuffers	entryBuffers
protected int	numThreads
protected ConcurrentHashMap<String,Long>	regionMaximumEditLogSeqNum
protected CancelableProgressable	reporter
protected AtomicLong	skippedEdits
protected List<org.apache.hadoop.fs.Path>	splits
protected ConcurrentHashMap<String,WALSplitter.SinkWriter>	writers
protected boolean	writersClosed
protected List<WALSplitter.WriterThread>	writerThreads

Constructor Summary

Constructors

Constructor and Description
OutputSink(WALSplitter.PipelineController controller, WALSplitter.EntryBuffers entryBuffers, int numWriters)

Method Summary

Modifier and Type	Method and Description
abstract void	append(WALSplitter.RegionEntryBuffer buffer)
protected boolean	finishWriting(boolean interrupt) Wait for writer threads to dump all info to the sink
abstract List<org.apache.hadoop.fs.Path>	finishWritingAndClose()
boolean	flush() WriterThread call this function to help flush internal remaining edits in buffer before close
abstract int	getNumberOfRecoveredRegions()
(package private) int	getNumOpenWriters()
abstract Map<byte[],Long>	getOutputCounts()
(package private) long	getSkippedEdits()
abstract boolean	keepRegionEvent(WAL.Entry entry) Some WALEdit's contain only KV's for account on what happened to a region.
(package private) void	setReporter(CancelableProgressable reporter)
void	startWriterThreads() Start the threads that will pump data from the entryBuffers to the output files.
(package private) void	updateRegionMaximumEditLogSeqNum(WAL.Entry entry) Update region's maximum edit log SeqNum.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

controller

protected WALSplitter.PipelineController controller

entryBuffers

protected WALSplitter.EntryBuffers entryBuffers

writers

protected ConcurrentHashMap<String,WALSplitter.SinkWriter> writers

regionMaximumEditLogSeqNum

protected final ConcurrentHashMap<String,Long> regionMaximumEditLogSeqNum

writerThreads

protected final List<WALSplitter.WriterThread> writerThreads

blacklistedRegions

protected final Set<byte[]> blacklistedRegions

closeAndCleanCompleted

protected boolean closeAndCleanCompleted

writersClosed

protected boolean writersClosed

numThreads

protected final int numThreads

reporter

protected CancelableProgressable reporter

skippedEdits

protected AtomicLong skippedEdits

splits

protected List<org.apache.hadoop.fs.Path> splits

Constructor Detail

OutputSink

public OutputSink(WALSplitter.PipelineController controller,
                  WALSplitter.EntryBuffers entryBuffers,
                  int numWriters)

Method Detail

setReporter

void setReporter(CancelableProgressable reporter)

startWriterThreads

public void startWriterThreads()

Start the threads that will pump data from the entryBuffers to the output files.

updateRegionMaximumEditLogSeqNum

void updateRegionMaximumEditLogSeqNum(WAL.Entry entry)

Update region's maximum edit log SeqNum.

getNumOpenWriters

int getNumOpenWriters()

Returns:

the number of currently opened writers

getSkippedEdits

long getSkippedEdits()

finishWriting

protected boolean finishWriting(boolean interrupt)
                         throws IOException

Wait for writer threads to dump all info to the sink

Returns:

true when there is no error

Throws:

IOException

finishWritingAndClose

public abstract List<org.apache.hadoop.fs.Path> finishWritingAndClose()
                                                               throws IOException

Throws:

IOException

getOutputCounts

public abstract Map<byte[],Long> getOutputCounts()

Returns:

a map from encoded region ID to the number of edits written out for that region.

getNumberOfRecoveredRegions

public abstract int getNumberOfRecoveredRegions()

Returns:

number of regions we've recovered

append

public abstract void append(WALSplitter.RegionEntryBuffer buffer)
                     throws IOException

Parameters:

buffer - A WAL Edit Entry

Throws:

IOException

flush

public boolean flush()
              throws IOException

WriterThread call this function to help flush internal remaining edits in buffer before close

Returns:

true when underlying sink has something to flush

Throws:

IOException

keepRegionEvent

public abstract boolean keepRegionEvent(WAL.Entry entry)

Some WALEdit's contain only KV's for account on what happened to a region. Not all sinks will want to get all of those edits.

Returns:

Return true if this sink wants to accept this region-level WALEdit.