Package org.apache.hadoop.hbase.wal

Class OutputSink

java.lang.Object

org.apache.hadoop.hbase.wal.OutputSink

Direct Known Subclasses:

AbstractRecoveredEditsOutputSink, BoundedRecoveredHFilesOutputSink

@Private
abstract class OutputSink
extends Object

The following class is an abstraction class to provide a common interface to support different ways of consuming recovered edits.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	OutputSink.WriterThread

Field Summary

Fields

Modifier and Type	Field	Description
protected final CompletionService<Void>	closeCompletionService
protected final ThreadPoolExecutor	closeThreadPool	Used when close this output sink.
private final WALSplitter.PipelineController	controller
protected final EntryBuffers	entryBuffers
private static final org.slf4j.Logger	LOG
protected final int	numThreads
protected CancelableProgressable	reporter
protected final List<org.apache.hadoop.fs.Path>	splits	List of all the files produced by this sink,
protected MonitoredTask	status
protected final AtomicLong	totalSkippedEdits
private final List<OutputSink.WriterThread>	writerThreads

Constructor Summary

Constructors

Constructor	Description
OutputSink(WALSplitter.PipelineController controller, EntryBuffers entryBuffers, int numWriters)

Method Summary

Modifier and Type	Method	Description
(package private) abstract void	append(EntryBuffers.RegionEntryBuffer buffer)
(package private) abstract List<org.apache.hadoop.fs.Path>	close()
(package private) boolean	finishWriterThreads()	Wait for writer threads to dump all info to the sink
(package private) abstract int	getNumberOfRecoveredRegions()	Returns number of regions we've recovered
(package private) abstract int	getNumOpenWriters()	Returns the number of currently opened writers
(package private) abstract Map<String,Long>	getOutputCounts()	Returns a map from encoded region ID to the number of edits written out for that region.
(package private) long	getTotalSkippedEdits()
(package private) abstract boolean	keepRegionEvent(WAL.Entry entry)	Some WALEdit's contain only KV's for account on what happened to a region.
(package private) void	setReporter(CancelableProgressable reporter)
(package private) void	setStatus(MonitoredTask status)
(package private) void	startWriterThreads()	Start the threads that will pump data from the entryBuffers to the output files.
protected final void	updateStatusWithMsg(String msg)	Set status message in MonitoredTask instance that is set in this OutputSink

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

LOG

private static final org.slf4j.Logger LOG

controller

private final WALSplitter.PipelineController controller

entryBuffers

protected final EntryBuffers entryBuffers

writerThreads

private final List<OutputSink.WriterThread> writerThreads

numThreads

protected final int numThreads

reporter

protected CancelableProgressable reporter

totalSkippedEdits

protected final AtomicLong totalSkippedEdits

splits

protected final List<org.apache.hadoop.fs.Path> splits

List of all the files produced by this sink,

Must be a synchronized list to avoid concurrency issues. CopyOnWriteArrayList is not a good choice because all we do is add to the list and then return the result.

status

protected MonitoredTask status

closeThreadPool

protected final ThreadPoolExecutor closeThreadPool

Used when close this output sink.

closeCompletionService

protected final CompletionService<Void> closeCompletionService

Constructor Details

OutputSink

public OutputSink(WALSplitter.PipelineController controller,
 EntryBuffers entryBuffers,
 int numWriters)

Method Details

setReporter

void setReporter(CancelableProgressable reporter)

setStatus

void setStatus(MonitoredTask status)

startWriterThreads

void startWriterThreads()
                 throws IOException

Start the threads that will pump data from the entryBuffers to the output files.

Throws:

IOException

finishWriterThreads

boolean finishWriterThreads()
                     throws IOException

Wait for writer threads to dump all info to the sink

Returns:

true when there is no error

Throws:

IOException

getTotalSkippedEdits

long getTotalSkippedEdits()

getNumOpenWriters

abstract int getNumOpenWriters()

Returns the number of currently opened writers

append

abstract void append(EntryBuffers.RegionEntryBuffer buffer)
              throws IOException

Parameters:

buffer - A buffer of some number of edits for a given region.

Throws:

IOException - For any IO errors

close

abstract List<org.apache.hadoop.fs.Path> close()
                                        throws IOException

Throws:

IOException

getOutputCounts

abstract Map<String,Long> getOutputCounts()

Returns a map from encoded region ID to the number of edits written out for that region.

getNumberOfRecoveredRegions

abstract int getNumberOfRecoveredRegions()

Returns number of regions we've recovered

keepRegionEvent

abstract boolean keepRegionEvent(WAL.Entry entry)

Some WALEdit's contain only KV's for account on what happened to a region. Not all sinks will want to get all of those edits.

Returns:

Return true if this sink wants to accept this region-level WALEdit.

updateStatusWithMsg

protected final void updateStatusWithMsg(String msg)

Set status message in MonitoredTask instance that is set in this OutputSink

Parameters:

msg - message to update the status with