Package org.apache.hadoop.hbase.wal

Class WALSplitter

java.lang.Object

org.apache.hadoop.hbase.wal.WALSplitter

@Private
public class WALSplitter
extends Object

Split RegionServer WAL files. Splits the WAL into new files, one per region, to be picked up on Region reopen. Deletes the split WAL when finished. Create an instance and call splitWAL(FileStatus, CancelableProgressable) per file or use static helper methods.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
(package private) static class	WALSplitter.CorruptedLogFileException
static class	WALSplitter.PipelineController	Contains some methods to control WAL-entries producer / consumer interactions
(package private) static final class	WALSplitter.SplitWALResult	Data structure returned as result by #splitWAL(FileStatus, CancelableProgressable).

Field Summary

Fields

Modifier and Type	Field	Description
private final long	bufferSize
protected final org.apache.hadoop.conf.Configuration	conf
static final boolean	DEFAULT_WAL_SPLIT_TO_HFILE
private EntryBuffers	entryBuffers
private org.apache.hadoop.fs.FileStatus	fileBeingSplit
private final boolean	hfile
protected Map<String,Long>	lastFlushedSequenceIds
private static final org.slf4j.Logger	LOG
private final int	numWriterThreads
(package private) OutputSink	outputSink
protected Map<String,Map<byte[],Long>>	regionMaxSeqIdInStores
(package private) final org.apache.hadoop.fs.Path	rootDir
(package private) final org.apache.hadoop.fs.FileSystem	rootFS
(package private) final RegionServerServices	rsServices
protected final LastSequenceId	sequenceIdChecker
private final boolean	skipErrors
static final boolean	SPLIT_SKIP_ERRORS_DEFAULT	By default we retry errors in splitting, rather than skipping.
static final String	SPLIT_SKIP_ERRORS_KEY
static final String	SPLIT_WAL_BUFFER_SIZE
static final String	SPLIT_WAL_WRITER_THREADS
static final String	SPLIT_WRITER_CREATION_BOUNDED	True if we are to run with bounded amount of writers rather than let the count blossom.
private SplitLogWorkerCoordination	splitLogWorkerCoordination	Coordinator for split log.
private final boolean	splitWriterCreationBounded
private final String	tmpDirName
static final String	WAL_SPLIT_TO_HFILE	Split WAL directly to hfiles instead of into intermediary 'recovered.edits' files.
private final WALFactory	walFactory
protected final org.apache.hadoop.fs.FileSystem	walFS
protected final org.apache.hadoop.fs.Path	walRootDir

Constructor Summary

Constructors

Constructor	Description
WALSplitter(WALFactory factory, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path walRootDir, org.apache.hadoop.fs.FileSystem walFS, org.apache.hadoop.fs.Path rootDir, org.apache.hadoop.fs.FileSystem rootFS)
WALSplitter(WALFactory factory, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path walRootDir, org.apache.hadoop.fs.FileSystem walFS, org.apache.hadoop.fs.Path rootDir, org.apache.hadoop.fs.FileSystem rootFS, LastSequenceId idChecker, SplitLogWorkerCoordination splitLogWorkerCoordination, RegionServerServices rsServices)

Method Summary

Modifier and Type	Method	Description
private void	createOutputSinkAndEntryBuffers()	Setup the output sinks and entry buffers ahead of splitting WAL.
protected WALProvider.Writer	createWriter(org.apache.hadoop.fs.Path logfile)	Create a new WALProvider.Writer for writing log splits.
(package private) org.apache.hadoop.fs.FileStatus	getFileBeingSplit()
private WAL.Entry	getNextLogLine(WALStreamReader in, org.apache.hadoop.fs.Path path, boolean skipErrors)
private int	getNumOpenWriters()	Get current open writers
protected WALStreamReader	getReader(org.apache.hadoop.fs.FileStatus walStatus, boolean skipErrors, CancelableProgressable cancel)	Create a new WALStreamReader for reading logs to split.
private WALStreamReader	getReader(org.apache.hadoop.fs.Path curLogFile, CancelableProgressable reporter)	Create a new WALStreamReader for reading logs to split.
(package private) Map<String,Map<byte[],Long>>	getRegionMaxSeqIdInStores()
(package private) String	getTmpDirName()
(package private) WALFactory	getWalFactory()
private boolean	isRegionDirPresentUnderRoot(TableName tn, String region)
static List<org.apache.hadoop.fs.Path>	split(org.apache.hadoop.fs.Path walRootDir, org.apache.hadoop.fs.Path walsDir, org.apache.hadoop.fs.Path archiveDir, org.apache.hadoop.fs.FileSystem walFS, org.apache.hadoop.conf.Configuration conf, WALFactory factory)	Split a folder of WAL files.
static boolean	splitLogFile(org.apache.hadoop.fs.Path walDir, org.apache.hadoop.fs.FileStatus logfile, org.apache.hadoop.fs.FileSystem walFS, org.apache.hadoop.conf.Configuration conf, CancelableProgressable reporter, LastSequenceId idChecker, SplitLogWorkerCoordination splitLogWorkerCoordination, WALFactory factory, RegionServerServices rsServices)	Splits a WAL file.
(package private) WALSplitter.SplitWALResult	splitWAL(org.apache.hadoop.fs.FileStatus walStatus, CancelableProgressable cancel)	WAL splitting implementation, splits one WAL file.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

LOG

private static final org.slf4j.Logger LOG

SPLIT_SKIP_ERRORS_KEY

public static final String SPLIT_SKIP_ERRORS_KEY

See Also:

• Constant Field Values

SPLIT_SKIP_ERRORS_DEFAULT

public static final boolean SPLIT_SKIP_ERRORS_DEFAULT

By default we retry errors in splitting, rather than skipping.

See Also:

• Constant Field Values

walRootDir

protected final org.apache.hadoop.fs.Path walRootDir

walFS

protected final org.apache.hadoop.fs.FileSystem walFS

conf

protected final org.apache.hadoop.conf.Configuration conf

rootDir

final org.apache.hadoop.fs.Path rootDir

rootFS

final org.apache.hadoop.fs.FileSystem rootFS

rsServices

final RegionServerServices rsServices

outputSink

OutputSink outputSink

entryBuffers

private EntryBuffers entryBuffers

splitLogWorkerCoordination

private SplitLogWorkerCoordination splitLogWorkerCoordination

Coordinator for split log. Used by the zk-based log splitter. Not used by the procedure v2-based log splitter.

walFactory

private final WALFactory walFactory

sequenceIdChecker

protected final LastSequenceId sequenceIdChecker

lastFlushedSequenceIds

protected Map<String,Long> lastFlushedSequenceIds

regionMaxSeqIdInStores

protected Map<String,Map<byte[],Long>> regionMaxSeqIdInStores

fileBeingSplit

private org.apache.hadoop.fs.FileStatus fileBeingSplit

tmpDirName

private final String tmpDirName

WAL_SPLIT_TO_HFILE

public static final String WAL_SPLIT_TO_HFILE

Split WAL directly to hfiles instead of into intermediary 'recovered.edits' files.

See Also:

• Constant Field Values

DEFAULT_WAL_SPLIT_TO_HFILE

public static final boolean DEFAULT_WAL_SPLIT_TO_HFILE

See Also:

• Constant Field Values

SPLIT_WRITER_CREATION_BOUNDED

public static final String SPLIT_WRITER_CREATION_BOUNDED

True if we are to run with bounded amount of writers rather than let the count blossom. Default is 'false'. Does not apply if you have set 'hbase.wal.split.to.hfile' as that is always bounded. Only applies when you are doing recovery to 'recovered.edits' files (the old default). Bounded writing tends to have higher throughput.

See Also:

• Constant Field Values

SPLIT_WAL_BUFFER_SIZE

public static final String SPLIT_WAL_BUFFER_SIZE

See Also:

• Constant Field Values

SPLIT_WAL_WRITER_THREADS

public static final String SPLIT_WAL_WRITER_THREADS

See Also:

• Constant Field Values

numWriterThreads

private final int numWriterThreads

bufferSize

private final long bufferSize

splitWriterCreationBounded

private final boolean splitWriterCreationBounded

hfile

private final boolean hfile

skipErrors

private final boolean skipErrors

Constructor Details

WALSplitter

WALSplitter(WALFactory factory,
 org.apache.hadoop.conf.Configuration conf,
 org.apache.hadoop.fs.Path walRootDir,
 org.apache.hadoop.fs.FileSystem walFS,
 org.apache.hadoop.fs.Path rootDir,
 org.apache.hadoop.fs.FileSystem rootFS)

WALSplitter

WALSplitter(WALFactory factory,
 org.apache.hadoop.conf.Configuration conf,
 org.apache.hadoop.fs.Path walRootDir,
 org.apache.hadoop.fs.FileSystem walFS,
 org.apache.hadoop.fs.Path rootDir,
 org.apache.hadoop.fs.FileSystem rootFS,
 LastSequenceId idChecker,
 SplitLogWorkerCoordination splitLogWorkerCoordination,
 RegionServerServices rsServices)

Method Details

getWalFactory

WALFactory getWalFactory()

getFileBeingSplit

org.apache.hadoop.fs.FileStatus getFileBeingSplit()

getTmpDirName

String getTmpDirName()

getRegionMaxSeqIdInStores

Map<String,Map<byte[],Long>> getRegionMaxSeqIdInStores()

splitLogFile

public static boolean splitLogFile(org.apache.hadoop.fs.Path walDir,
 org.apache.hadoop.fs.FileStatus logfile,
 org.apache.hadoop.fs.FileSystem walFS,
 org.apache.hadoop.conf.Configuration conf,
 CancelableProgressable reporter,
 LastSequenceId idChecker,
 SplitLogWorkerCoordination splitLogWorkerCoordination,
 WALFactory factory,
 RegionServerServices rsServices)
                            throws IOException

Splits a WAL file. Used by old SplitLogWorker and tests. Not used by new procedure-based WAL splitter.

Returns:

false if it is interrupted by the progress-able.

Throws:

IOException

split

public static List<org.apache.hadoop.fs.Path> split(org.apache.hadoop.fs.Path walRootDir,
 org.apache.hadoop.fs.Path walsDir,
 org.apache.hadoop.fs.Path archiveDir,
 org.apache.hadoop.fs.FileSystem walFS,
 org.apache.hadoop.conf.Configuration conf,
 WALFactory factory)
                                             throws IOException

Split a folder of WAL files. Delete the directory when done. Used by tools and unit tests. It should be package private. It is public only because TestWALObserver is in a different package, which uses this method to do log splitting.

Returns:

List of output files created by the split.

Throws:

IOException

createOutputSinkAndEntryBuffers

private void createOutputSinkAndEntryBuffers()

Setup the output sinks and entry buffers ahead of splitting WAL.

splitWAL

WALSplitter.SplitWALResult splitWAL(org.apache.hadoop.fs.FileStatus walStatus,
 CancelableProgressable cancel)
                             throws IOException

WAL splitting implementation, splits one WAL file.

Parameters:

walStatus - should be for an actual WAL file.

Throws:

IOException

isRegionDirPresentUnderRoot

private boolean isRegionDirPresentUnderRoot(TableName tn,
 String region)
                                     throws IOException

Throws:

IOException

getReader

protected WALStreamReader getReader(org.apache.hadoop.fs.FileStatus walStatus,
 boolean skipErrors,
 CancelableProgressable cancel)
                             throws IOException,
WALSplitter.CorruptedLogFileException

Create a new WALStreamReader for reading logs to split.

Returns:

Returns null if file has length zero or file can't be found.

Throws:

IOException

WALSplitter.CorruptedLogFileException

getNextLogLine

private WAL.Entry getNextLogLine(WALStreamReader in,
 org.apache.hadoop.fs.Path path,
 boolean skipErrors)
                          throws WALSplitter.CorruptedLogFileException,
IOException

Throws:

WALSplitter.CorruptedLogFileException

IOException

createWriter

protected WALProvider.Writer createWriter(org.apache.hadoop.fs.Path logfile)
                                   throws IOException

Create a new WALProvider.Writer for writing log splits.

Returns:

a new Writer instance, caller should close

Throws:

IOException

getReader

private WALStreamReader getReader(org.apache.hadoop.fs.Path curLogFile,
 CancelableProgressable reporter)
                           throws IOException

Create a new WALStreamReader for reading logs to split.

Returns:

new Reader instance, caller should close

Throws:

IOException

getNumOpenWriters

private int getNumOpenWriters()

Get current open writers