Package org.apache.hadoop.hbase.wal

Class OutputSink

java.lang.Object

org.apache.hadoop.hbase.wal.OutputSink

Direct Known Subclasses:

AbstractRecoveredEditsOutputSink, BoundedRecoveredHFilesOutputSink, RegionReplicaReplicationEndpoint.RegionReplicaOutputSink

@Private
public abstract class OutputSink
extends Object

The following class is an abstraction class to provide a common interface to support different ways of consuming recovered edits.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	OutputSink.WriterThread

Field Summary

Fields

Modifier and Type	Field	Description
protected final CompletionService<Void>	closeCompletionService
protected final ThreadPoolExecutor	closeThreadPool	Used when close this output sink.
private final WALSplitter.PipelineController	controller
protected final EntryBuffers	entryBuffers
private static final org.slf4j.Logger	LOG
protected final int	numThreads
protected CancelableProgressable	reporter
protected final List<org.apache.hadoop.fs.Path>	splits	List of all the files produced by this sink
protected MonitoredTask	status
protected final AtomicLong	totalSkippedEdits
private final List<OutputSink.WriterThread>	writerThreads

Constructor Summary

Constructors

Constructor	Description
OutputSink(WALSplitter.PipelineController controller, EntryBuffers entryBuffers, int numWriters)

Method Summary

Modifier and Type	Method	Description
protected abstract void	append(EntryBuffers.RegionEntryBuffer buffer)
protected abstract List<org.apache.hadoop.fs.Path>	close()
protected boolean	finishWriterThreads(boolean interrupt)	Wait for writer threads to dump all info to the sink
protected abstract int	getNumberOfRecoveredRegions()	Returns number of regions we've recovered
protected abstract int	getNumOpenWriters()	Returns the number of currently opened writers
protected abstract Map<String,Long>	getOutputCounts()	Returns a map from encoded region ID to the number of edits written out for that region.
(package private) long	getTotalSkippedEdits()
protected abstract boolean	keepRegionEvent(WAL.Entry entry)	Some WALEdit's contain only KV's for account on what happened to a region.
void	restartWriterThreadsIfNeeded()
(package private) void	setReporter(CancelableProgressable reporter)
(package private) void	setStatus(MonitoredTask status)
void	startWriterThreads()	Start the threads that will pump data from the entryBuffers to the output files.
protected final void	updateStatusWithMsg(String msg)	Set status message in MonitoredTask instance that is set in this OutputSink

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

LOG

private static final org.slf4j.Logger LOG

controller

private final WALSplitter.PipelineController controller

entryBuffers

protected final EntryBuffers entryBuffers

writerThreads

private final List<OutputSink.WriterThread> writerThreads

numThreads

protected final int numThreads

reporter

protected CancelableProgressable reporter

totalSkippedEdits

protected final AtomicLong totalSkippedEdits

splits

protected final List<org.apache.hadoop.fs.Path> splits

List of all the files produced by this sink

status

protected MonitoredTask status

closeThreadPool

protected final ThreadPoolExecutor closeThreadPool

Used when close this output sink.

closeCompletionService

protected final CompletionService<Void> closeCompletionService

Constructor Details

OutputSink

public OutputSink(WALSplitter.PipelineController controller,
 EntryBuffers entryBuffers,
 int numWriters)

Method Details

setReporter

void setReporter(CancelableProgressable reporter)

setStatus

void setStatus(MonitoredTask status)

startWriterThreads

public void startWriterThreads()
                        throws IOException

Start the threads that will pump data from the entryBuffers to the output files.

Throws:

IOException

restartWriterThreadsIfNeeded

public void restartWriterThreadsIfNeeded()

finishWriterThreads

protected boolean finishWriterThreads(boolean interrupt)
                               throws IOException

Wait for writer threads to dump all info to the sink

Returns:

true when there is no error

Throws:

IOException

getTotalSkippedEdits

long getTotalSkippedEdits()

getNumOpenWriters

protected abstract int getNumOpenWriters()

Returns the number of currently opened writers

append

protected abstract void append(EntryBuffers.RegionEntryBuffer buffer)
                        throws IOException

Parameters:

buffer - A buffer of some number of edits for a given region.

Throws:

IOException - For any IO errors

close

protected abstract List<org.apache.hadoop.fs.Path> close()
                                                  throws IOException

Throws:

IOException

getOutputCounts

protected abstract Map<String,Long> getOutputCounts()

Returns a map from encoded region ID to the number of edits written out for that region.

getNumberOfRecoveredRegions

protected abstract int getNumberOfRecoveredRegions()

Returns number of regions we've recovered

keepRegionEvent

protected abstract boolean keepRegionEvent(WAL.Entry entry)

Some WALEdit's contain only KV's for account on what happened to a region. Not all sinks will want to get all of those edits.

Returns:

Return true if this sink wants to accept this region-level WALEdit.

updateStatusWithMsg

protected final void updateStatusWithMsg(String msg)

Set status message in MonitoredTask instance that is set in this OutputSink

Parameters:

msg - message to update the status with