org.apache.hadoop.hbase.wal

Class WALSplitter

java.lang.Object

org.apache.hadoop.hbase.wal.WALSplitter

@InterfaceAudience.Private
public class WALSplitter
extends Object

This class is responsible for splitting up a bunch of regionserver commit log files that are no longer being written to, into new files, one per region, for recovering data on startup. Delete the old log files when finished.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
(package private) static class	WALSplitter.CorruptedLogFileException
static class	WALSplitter.PipelineController Contains some methods to control WAL-entries producer / consumer interactions
static class	WALSplitter.RegionEntryBuffer A buffer of some number of edits for a given region.
static class	WALSplitter.SinkWriter Class wraps the actual writer which writes data out and related statistics
(package private) static class	WALSplitter.WriterAndPath Private data structure that wraps a Writer and its Path, also collecting statistics about the data written to this output.

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.hadoop.conf.Configuration	conf
private EntryBuffers	entryBuffers
private org.apache.hadoop.fs.FileStatus	fileBeingSplit
protected Map<String,Long>	lastFlushedSequenceIds
private static org.slf4j.Logger	LOG
(package private) OutputSink	outputSink
protected Map<String,Map<byte[],Long>>	regionMaxSeqIdInStores
protected LastSequenceId	sequenceIdChecker
static boolean	SPLIT_SKIP_ERRORS_DEFAULT By default we retry errors in splitting, rather than skipping.
static String	SPLIT_WRITER_CREATION_BOUNDED
private SplitLogWorkerCoordination	splitLogWorkerCoordination
private boolean	splitWriterCreationBounded
private MonitoredTask	status
protected org.apache.hadoop.fs.Path	walDir
private WALFactory	walFactory
protected org.apache.hadoop.fs.FileSystem	walFS

Constructor Summary

Constructors

Constructor and Description
WALSplitter(WALFactory factory, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path walDir, org.apache.hadoop.fs.FileSystem walFS, LastSequenceId idChecker, SplitLogWorkerCoordination splitLogWorkerCoordination)

Method Summary

Modifier and Type	Method and Description
protected WALProvider.Writer	createWriter(org.apache.hadoop.fs.Path logfile) Create a new WALProvider.Writer for writing log splits.
(package private) org.apache.hadoop.fs.FileStatus	getFileBeingSplit()
private static WAL.Entry	getNextLogLine(WAL.Reader in, org.apache.hadoop.fs.Path path, boolean skipErrors)
private int	getNumOpenWriters() Get current open writers
protected WAL.Reader	getReader(org.apache.hadoop.fs.FileStatus file, boolean skipErrors, CancelableProgressable reporter) Create a new WAL.Reader for reading logs to split.
protected WAL.Reader	getReader(org.apache.hadoop.fs.Path curLogFile, CancelableProgressable reporter) Create a new WAL.Reader for reading logs to split.
(package private) Map<String,Map<byte[],Long>>	getRegionMaxSeqIdInStores()
(package private) WALFactory	getWalFactory()
static List<org.apache.hadoop.fs.Path>	split(org.apache.hadoop.fs.Path rootDir, org.apache.hadoop.fs.Path logDir, org.apache.hadoop.fs.Path oldLogDir, org.apache.hadoop.fs.FileSystem walFS, org.apache.hadoop.conf.Configuration conf, WALFactory factory)
(package private) boolean	splitLogFile(org.apache.hadoop.fs.FileStatus logfile, CancelableProgressable reporter) log splitting implementation, splits one log file.
static boolean	splitLogFile(org.apache.hadoop.fs.Path walDir, org.apache.hadoop.fs.FileStatus logfile, org.apache.hadoop.fs.FileSystem walFS, org.apache.hadoop.conf.Configuration conf, CancelableProgressable reporter, LastSequenceId idChecker, SplitLogWorkerCoordination splitLogWorkerCoordination, WALFactory factory) Splits a WAL file into region's recovered-edits directory.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

private static final org.slf4j.Logger LOG

SPLIT_SKIP_ERRORS_DEFAULT

public static final boolean SPLIT_SKIP_ERRORS_DEFAULT

By default we retry errors in splitting, rather than skipping.

See Also:

Constant Field Values

walDir

protected final org.apache.hadoop.fs.Path walDir

walFS

protected final org.apache.hadoop.fs.FileSystem walFS

conf

protected final org.apache.hadoop.conf.Configuration conf

outputSink

OutputSink outputSink

entryBuffers

private EntryBuffers entryBuffers

splitLogWorkerCoordination

private SplitLogWorkerCoordination splitLogWorkerCoordination

walFactory

private final WALFactory walFactory

status

private MonitoredTask status

sequenceIdChecker

protected final LastSequenceId sequenceIdChecker

lastFlushedSequenceIds

protected Map<String,Long> lastFlushedSequenceIds

regionMaxSeqIdInStores

protected Map<String,Map<byte[],Long>> regionMaxSeqIdInStores

fileBeingSplit

private org.apache.hadoop.fs.FileStatus fileBeingSplit

splitWriterCreationBounded

private final boolean splitWriterCreationBounded

SPLIT_WRITER_CREATION_BOUNDED

public static final String SPLIT_WRITER_CREATION_BOUNDED

See Also:

Constant Field Values

Constructor Detail

WALSplitter

WALSplitter(WALFactory factory,
            org.apache.hadoop.conf.Configuration conf,
            org.apache.hadoop.fs.Path walDir,
            org.apache.hadoop.fs.FileSystem walFS,
            LastSequenceId idChecker,
            SplitLogWorkerCoordination splitLogWorkerCoordination)

Method Detail

getWalFactory

WALFactory getWalFactory()

getFileBeingSplit

org.apache.hadoop.fs.FileStatus getFileBeingSplit()

getRegionMaxSeqIdInStores

Map<String,Map<byte[],Long>> getRegionMaxSeqIdInStores()

splitLogFile

public static boolean splitLogFile(org.apache.hadoop.fs.Path walDir,
                                   org.apache.hadoop.fs.FileStatus logfile,
                                   org.apache.hadoop.fs.FileSystem walFS,
                                   org.apache.hadoop.conf.Configuration conf,
                                   CancelableProgressable reporter,
                                   LastSequenceId idChecker,
                                   SplitLogWorkerCoordination splitLogWorkerCoordination,
                                   WALFactory factory)
                            throws IOException

Splits a WAL file into region's recovered-edits directory. This is the main entry point for distributed log splitting from SplitLogWorker.

If the log file has N regions then N recovered.edits files will be produced.

Returns:

false if it is interrupted by the progress-able.

Throws:

IOException

split

public static List<org.apache.hadoop.fs.Path> split(org.apache.hadoop.fs.Path rootDir,
                                                    org.apache.hadoop.fs.Path logDir,
                                                    org.apache.hadoop.fs.Path oldLogDir,
                                                    org.apache.hadoop.fs.FileSystem walFS,
                                                    org.apache.hadoop.conf.Configuration conf,
                                                    WALFactory factory)
                                             throws IOException

Throws:

IOException

splitLogFile

boolean splitLogFile(org.apache.hadoop.fs.FileStatus logfile,
                     CancelableProgressable reporter)
              throws IOException

log splitting implementation, splits one log file.

Parameters:

logfile - should be an actual log file.

Throws:

IOException

getReader

protected WAL.Reader getReader(org.apache.hadoop.fs.FileStatus file,
                               boolean skipErrors,
                               CancelableProgressable reporter)
                        throws IOException,
                               WALSplitter.CorruptedLogFileException

Create a new WAL.Reader for reading logs to split.

Throws:

IOException

WALSplitter.CorruptedLogFileException

getNextLogLine

private static WAL.Entry getNextLogLine(WAL.Reader in,
                                        org.apache.hadoop.fs.Path path,
                                        boolean skipErrors)
                                 throws WALSplitter.CorruptedLogFileException,
                                        IOException

Throws:

WALSplitter.CorruptedLogFileException

IOException

createWriter

protected WALProvider.Writer createWriter(org.apache.hadoop.fs.Path logfile)
                                   throws IOException

Create a new WALProvider.Writer for writing log splits.

Returns:

a new Writer instance, caller should close

Throws:

IOException

getReader

protected WAL.Reader getReader(org.apache.hadoop.fs.Path curLogFile,
                               CancelableProgressable reporter)
                        throws IOException

Create a new WAL.Reader for reading logs to split.

Returns:

new Reader instance, caller should close

Throws:

IOException

getNumOpenWriters

private int getNumOpenWriters()

Get current open writers