org.apache.hadoop.hbase.wal

Class WALSplitter

java.lang.Object

org.apache.hadoop.hbase.wal.WALSplitter

@InterfaceAudience.Private
public class WALSplitter
extends Object

This class is responsible for splitting up a bunch of regionserver commit log files that are no longer being written to, into new files, one per region, for recovering data on startup. Delete the old log files when finished.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
(package private) class	WALSplitter.BoundedLogWriterCreationOutputSink
(package private) static class	WALSplitter.CorruptedLogFileException
static class	WALSplitter.EntryBuffers Class which accumulates edits and separates them into a buffer per region while simultaneously accounting RAM usage.
(package private) class	WALSplitter.LogRecoveredEditsOutputSink Class that manages the output streams from the log splitting process.
static class	WALSplitter.MutationReplay A struct used by getMutationsFromWALEntry
static class	WALSplitter.OutputSink The following class is an abstraction class to provide a common interface to support different ways of consuming recovered edits.
static class	WALSplitter.PipelineController Contains some methods to control WAL-entries producer / consumer interactions
static class	WALSplitter.RegionEntryBuffer A buffer of some number of edits for a given region.
static class	WALSplitter.SinkWriter Class wraps the actual writer which writes data out and related statistics
private static class	WALSplitter.WriterAndPath Private data structure that wraps a Writer and its Path, also collecting statistics about the data written to this output.
static class	WALSplitter.WriterThread

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.hadoop.conf.Configuration	conf
private static Pattern	EDITFILES_NAME_PATTERN
private WALSplitter.EntryBuffers	entryBuffers
private org.apache.hadoop.fs.FileStatus	fileBeingSplit
protected Map<String,Long>	lastFlushedSequenceIds
private static org.slf4j.Logger	LOG
private static String	OLD_SEQUENCE_ID_FILE_SUFFIX
(package private) WALSplitter.OutputSink	outputSink
private static String	RECOVERED_LOG_TMPFILE_SUFFIX
protected Map<String,Map<byte[],Long>>	regionMaxSeqIdInStores
private static String	SEQUENCE_ID_FILE_SUFFIX
private static int	SEQUENCE_ID_FILE_SUFFIX_LENGTH
protected LastSequenceId	sequenceIdChecker
static boolean	SPLIT_SKIP_ERRORS_DEFAULT By default we retry errors in splitting, rather than skipping.
static String	SPLIT_WRITER_CREATION_BOUNDED
private SplitLogWorkerCoordination	splitLogWorkerCoordination
private boolean	splitWriterCreationBounded
private MonitoredTask	status
protected org.apache.hadoop.fs.Path	walDir
private WALFactory	walFactory
protected org.apache.hadoop.fs.FileSystem	walFS

Constructor Summary

Constructors

Constructor and Description
WALSplitter(WALFactory factory, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path walDir, org.apache.hadoop.fs.FileSystem walFS, LastSequenceId idChecker, SplitLogWorkerCoordination splitLogWorkerCoordination)

Method Summary

Modifier and Type	Method and Description
private static void	archiveLogs(List<org.apache.hadoop.fs.Path> corruptedLogs, List<org.apache.hadoop.fs.Path> processedLogs, org.apache.hadoop.fs.Path oldLogDir, org.apache.hadoop.fs.FileSystem walFS, org.apache.hadoop.conf.Configuration conf) Moves processed logs to a oldLogDir after successful processing Moves corrupted logs (any log that couldn't be successfully parsed to corruptDir (.corrupt) for later investigation
protected WALProvider.Writer	createWriter(org.apache.hadoop.fs.Path logfile) Create a new WALProvider.Writer for writing log splits.
private static void	finishSplitLogFile(org.apache.hadoop.fs.Path walDir, org.apache.hadoop.fs.Path oldLogDir, org.apache.hadoop.fs.Path logPath, org.apache.hadoop.conf.Configuration conf)
static void	finishSplitLogFile(String logfile, org.apache.hadoop.conf.Configuration conf) Completes the work done by splitLogFile by archiving logs
(package private) static String	formatRecoveredEditsFileName(long seqid)
private static org.apache.hadoop.fs.Path	getCompletedRecoveredEditsFilePath(org.apache.hadoop.fs.Path srcPath, long maximumEditLogSeqNum) Get the completed recovered edits file path, renaming it to be by last edit in the file from its first edit.
static long	getMaxRegionSequenceId(org.apache.hadoop.conf.Configuration conf, RegionInfo region, CollectionUtils.IOExceptionSupplier<org.apache.hadoop.fs.FileSystem> rootFsSupplier, CollectionUtils.IOExceptionSupplier<org.apache.hadoop.fs.FileSystem> walFsSupplier) Deprecated. Only for compatibility, will be removed in 4.0.0.
static long	getMaxRegionSequenceId(org.apache.hadoop.fs.FileSystem walFS, org.apache.hadoop.fs.Path regionDir) Get the max sequence id which is stored in the region directory.
private static long	getMaxSequenceId(org.apache.hadoop.fs.FileStatus[] files)
static List<WALSplitter.MutationReplay>	getMutationsFromWALEntry(org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos.WALEntry entry, CellScanner cells, Pair<WALKey,WALEdit> logEntry, Durability durability) This function is used to construct mutations from a WALEntry.
private static WAL.Entry	getNextLogLine(WAL.Reader in, org.apache.hadoop.fs.Path path, boolean skipErrors)
private int	getNumOpenWriters() Get current open writers
protected WAL.Reader	getReader(org.apache.hadoop.fs.FileStatus file, boolean skipErrors, CancelableProgressable reporter) Create a new WAL.Reader for reading logs to split.
protected WAL.Reader	getReader(org.apache.hadoop.fs.Path curLogFile, CancelableProgressable reporter) Create a new WAL.Reader for reading logs to split.
static org.apache.hadoop.fs.Path	getRegionDirRecoveredEditsDir(org.apache.hadoop.fs.Path regionDir)
(package private) static org.apache.hadoop.fs.Path	getRegionSplitEditsPath(WAL.Entry logEntry, String fileNameBeingSplit, org.apache.hadoop.conf.Configuration conf) Path to a file under RECOVERED_EDITS_DIR directory of the region found in logEntry named for the sequenceid in the passed logEntry: e.g.
private static org.apache.hadoop.fs.FileStatus[]	getSequenceIdFiles(org.apache.hadoop.fs.FileSystem walFS, org.apache.hadoop.fs.Path regionDir)
static NavigableSet<org.apache.hadoop.fs.Path>	getSplitEditFilesSorted(org.apache.hadoop.fs.FileSystem walFS, org.apache.hadoop.fs.Path regionDir) Returns sorted set of edit files made by splitter, excluding files with '.temp' suffix.
private static String	getTmpRecoveredEditsFileName(String fileName)
static boolean	hasRecoveredEdits(org.apache.hadoop.conf.Configuration conf, RegionInfo regionInfo) Check whether there is recovered.edits in the region dir
static boolean	isSequenceIdFile(org.apache.hadoop.fs.Path file) Is the given file a region open sequence id file.
static org.apache.hadoop.fs.Path	moveAsideBadEditsFile(org.apache.hadoop.fs.FileSystem walFS, org.apache.hadoop.fs.Path edits) Move aside a bad edits file.
static List<org.apache.hadoop.fs.Path>	split(org.apache.hadoop.fs.Path rootDir, org.apache.hadoop.fs.Path logDir, org.apache.hadoop.fs.Path oldLogDir, org.apache.hadoop.fs.FileSystem walFS, org.apache.hadoop.conf.Configuration conf, WALFactory factory)
(package private) boolean	splitLogFile(org.apache.hadoop.fs.FileStatus logfile, CancelableProgressable reporter) log splitting implementation, splits one log file.
static boolean	splitLogFile(org.apache.hadoop.fs.Path walDir, org.apache.hadoop.fs.FileStatus logfile, org.apache.hadoop.fs.FileSystem walFS, org.apache.hadoop.conf.Configuration conf, CancelableProgressable reporter, LastSequenceId idChecker, SplitLogWorkerCoordination splitLogWorkerCoordination, WALFactory factory) Splits a WAL file into region's recovered-edits directory.
static void	writeRegionSequenceIdFile(org.apache.hadoop.fs.FileSystem walFS, org.apache.hadoop.fs.Path regionDir, long newMaxSeqId) Create a file with name as region's max sequence id

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

private static final org.slf4j.Logger LOG

SPLIT_SKIP_ERRORS_DEFAULT

public static final boolean SPLIT_SKIP_ERRORS_DEFAULT

By default we retry errors in splitting, rather than skipping.

See Also:

Constant Field Values

walDir

protected final org.apache.hadoop.fs.Path walDir

walFS

protected final org.apache.hadoop.fs.FileSystem walFS

conf

protected final org.apache.hadoop.conf.Configuration conf

outputSink

WALSplitter.OutputSink outputSink

entryBuffers

private WALSplitter.EntryBuffers entryBuffers

splitLogWorkerCoordination

private SplitLogWorkerCoordination splitLogWorkerCoordination

walFactory

private final WALFactory walFactory

status

private MonitoredTask status

sequenceIdChecker

protected final LastSequenceId sequenceIdChecker

lastFlushedSequenceIds

protected Map<String,Long> lastFlushedSequenceIds

regionMaxSeqIdInStores

protected Map<String,Map<byte[],Long>> regionMaxSeqIdInStores

fileBeingSplit

private org.apache.hadoop.fs.FileStatus fileBeingSplit

splitWriterCreationBounded

private final boolean splitWriterCreationBounded

SPLIT_WRITER_CREATION_BOUNDED

public static final String SPLIT_WRITER_CREATION_BOUNDED

See Also:

Constant Field Values

EDITFILES_NAME_PATTERN

private static final Pattern EDITFILES_NAME_PATTERN

RECOVERED_LOG_TMPFILE_SUFFIX

private static final String RECOVERED_LOG_TMPFILE_SUFFIX

See Also:

Constant Field Values

SEQUENCE_ID_FILE_SUFFIX

private static final String SEQUENCE_ID_FILE_SUFFIX

See Also:

Constant Field Values

OLD_SEQUENCE_ID_FILE_SUFFIX

private static final String OLD_SEQUENCE_ID_FILE_SUFFIX

See Also:

Constant Field Values

SEQUENCE_ID_FILE_SUFFIX_LENGTH

private static final int SEQUENCE_ID_FILE_SUFFIX_LENGTH

Constructor Detail

WALSplitter

WALSplitter(WALFactory factory,
            org.apache.hadoop.conf.Configuration conf,
            org.apache.hadoop.fs.Path walDir,
            org.apache.hadoop.fs.FileSystem walFS,
            LastSequenceId idChecker,
            SplitLogWorkerCoordination splitLogWorkerCoordination)

Method Detail

splitLogFile

public static boolean splitLogFile(org.apache.hadoop.fs.Path walDir,
                                   org.apache.hadoop.fs.FileStatus logfile,
                                   org.apache.hadoop.fs.FileSystem walFS,
                                   org.apache.hadoop.conf.Configuration conf,
                                   CancelableProgressable reporter,
                                   LastSequenceId idChecker,
                                   SplitLogWorkerCoordination splitLogWorkerCoordination,
                                   WALFactory factory)
                            throws IOException

Splits a WAL file into region's recovered-edits directory. This is the main entry point for distributed log splitting from SplitLogWorker.

If the log file has N regions then N recovered.edits files will be produced.

Returns:

false if it is interrupted by the progress-able.

Throws:

IOException

split

public static List<org.apache.hadoop.fs.Path> split(org.apache.hadoop.fs.Path rootDir,
                                                    org.apache.hadoop.fs.Path logDir,
                                                    org.apache.hadoop.fs.Path oldLogDir,
                                                    org.apache.hadoop.fs.FileSystem walFS,
                                                    org.apache.hadoop.conf.Configuration conf,
                                                    WALFactory factory)
                                             throws IOException

Throws:

IOException

splitLogFile

boolean splitLogFile(org.apache.hadoop.fs.FileStatus logfile,
                     CancelableProgressable reporter)
              throws IOException

log splitting implementation, splits one log file.

Parameters:

logfile - should be an actual log file.

Throws:

IOException

finishSplitLogFile

public static void finishSplitLogFile(String logfile,
                                      org.apache.hadoop.conf.Configuration conf)
                               throws IOException

Completes the work done by splitLogFile by archiving logs

It is invoked by SplitLogManager once it knows that one of the SplitLogWorkers have completed the splitLogFile() part. If the master crashes then this function might get called multiple times.

Parameters:

logfile -

conf -

Throws:

IOException

finishSplitLogFile

private static void finishSplitLogFile(org.apache.hadoop.fs.Path walDir,
                                       org.apache.hadoop.fs.Path oldLogDir,
                                       org.apache.hadoop.fs.Path logPath,
                                       org.apache.hadoop.conf.Configuration conf)
                                throws IOException

Throws:

IOException

archiveLogs

private static void archiveLogs(List<org.apache.hadoop.fs.Path> corruptedLogs,
                                List<org.apache.hadoop.fs.Path> processedLogs,
                                org.apache.hadoop.fs.Path oldLogDir,
                                org.apache.hadoop.fs.FileSystem walFS,
                                org.apache.hadoop.conf.Configuration conf)
                         throws IOException

Moves processed logs to a oldLogDir after successful processing Moves corrupted logs (any log that couldn't be successfully parsed to corruptDir (.corrupt) for later investigation

Parameters:

corruptedLogs -

processedLogs -

oldLogDir -

walFS - WAL FileSystem to archive files on.

conf -

Throws:

IOException

getRegionSplitEditsPath

static org.apache.hadoop.fs.Path getRegionSplitEditsPath(WAL.Entry logEntry,
                                                         String fileNameBeingSplit,
                                                         org.apache.hadoop.conf.Configuration conf)
                                                  throws IOException

Path to a file under RECOVERED_EDITS_DIR directory of the region found in logEntry named for the sequenceid in the passed logEntry: e.g. /hbase/some_table/2323432434/recovered.edits/2332. This method also ensures existence of RECOVERED_EDITS_DIR under the region creating it if necessary.

Parameters:

fs -

logEntry -

rootDir - HBase root dir.

fileNameBeingSplit - the file being split currently. Used to generate tmp file name.

Returns:

Path to file into which to dump split log edits.

Throws:

IOException

getTmpRecoveredEditsFileName

private static String getTmpRecoveredEditsFileName(String fileName)

getCompletedRecoveredEditsFilePath

private static org.apache.hadoop.fs.Path getCompletedRecoveredEditsFilePath(org.apache.hadoop.fs.Path srcPath,
                                                                            long maximumEditLogSeqNum)

Get the completed recovered edits file path, renaming it to be by last edit in the file from its first edit. Then we could use the name to skip recovered edits when doing HRegion.replayRecoveredEditsIfAny(java.util.Map<byte[], java.lang.Long>, org.apache.hadoop.hbase.util.CancelableProgressable, org.apache.hadoop.hbase.monitoring.MonitoredTask).

Parameters:

srcPath -

maximumEditLogSeqNum -

Returns:

dstPath take file's last edit log seq num as the name

formatRecoveredEditsFileName

static String formatRecoveredEditsFileName(long seqid)

getRegionDirRecoveredEditsDir

public static org.apache.hadoop.fs.Path getRegionDirRecoveredEditsDir(org.apache.hadoop.fs.Path regionDir)

Parameters:

regionDir - This regions directory in the filesystem.

Returns:

The directory that holds recovered edits files for the region regionDir

hasRecoveredEdits

public static boolean hasRecoveredEdits(org.apache.hadoop.conf.Configuration conf,
                                        RegionInfo regionInfo)
                                 throws IOException

Check whether there is recovered.edits in the region dir

Parameters:

conf - conf

regionInfo - the region to check

Returns:

true if recovered.edits exist in the region dir

Throws:

IOException - IOException

getSplitEditFilesSorted

public static NavigableSet<org.apache.hadoop.fs.Path> getSplitEditFilesSorted(org.apache.hadoop.fs.FileSystem walFS,
                                                                              org.apache.hadoop.fs.Path regionDir)
                                                                       throws IOException

Returns sorted set of edit files made by splitter, excluding files with '.temp' suffix.

Parameters:

walFS - WAL FileSystem used to retrieving split edits files.

regionDir - WAL region dir to look for recovered edits files under.

Returns:

Files in passed regionDir as a sorted set.

Throws:

IOException

moveAsideBadEditsFile

public static org.apache.hadoop.fs.Path moveAsideBadEditsFile(org.apache.hadoop.fs.FileSystem walFS,
                                                              org.apache.hadoop.fs.Path edits)
                                                       throws IOException

Move aside a bad edits file.

Parameters:

walFS - WAL FileSystem used to rename bad edits file.

edits - Edits file to move aside.

Returns:

The name of the moved aside file.

Throws:

IOException

isSequenceIdFile

public static boolean isSequenceIdFile(org.apache.hadoop.fs.Path file)

Is the given file a region open sequence id file.

getSequenceIdFiles

private static org.apache.hadoop.fs.FileStatus[] getSequenceIdFiles(org.apache.hadoop.fs.FileSystem walFS,
                                                                    org.apache.hadoop.fs.Path regionDir)
                                                             throws IOException

Throws:

IOException

getMaxSequenceId

private static long getMaxSequenceId(org.apache.hadoop.fs.FileStatus[] files)

getMaxRegionSequenceId

public static long getMaxRegionSequenceId(org.apache.hadoop.fs.FileSystem walFS,
                                          org.apache.hadoop.fs.Path regionDir)
                                   throws IOException

Get the max sequence id which is stored in the region directory. -1 if none.

Throws:

IOException

writeRegionSequenceIdFile

public static void writeRegionSequenceIdFile(org.apache.hadoop.fs.FileSystem walFS,
                                             org.apache.hadoop.fs.Path regionDir,
                                             long newMaxSeqId)
                                      throws IOException

Create a file with name as region's max sequence id

Throws:

IOException

getMaxRegionSequenceId

@Deprecated
public static long getMaxRegionSequenceId(org.apache.hadoop.conf.Configuration conf,
                                                      RegionInfo region,
                                                      CollectionUtils.IOExceptionSupplier<org.apache.hadoop.fs.FileSystem> rootFsSupplier,
                                                      CollectionUtils.IOExceptionSupplier<org.apache.hadoop.fs.FileSystem> walFsSupplier)
                                               throws IOException

Deprecated. Only for compatibility, will be removed in 4.0.0.

This method will check 3 places for finding the max sequence id file. One is the expected place, another is the old place under the region directory, and the last one is the wrong one we introduced in HBASE-20734. See HBASE-22617 for more details.

Notice that, you should always call this method instead of getMaxRegionSequenceId(FileSystem, Path) until 4.0.0 release.

Throws:

IOException

getReader

protected WAL.Reader getReader(org.apache.hadoop.fs.FileStatus file,
                               boolean skipErrors,
                               CancelableProgressable reporter)
                        throws IOException,
                               WALSplitter.CorruptedLogFileException

Create a new WAL.Reader for reading logs to split.

Parameters:

file -

Returns:

A new Reader instance, caller should close

Throws:

IOException

WALSplitter.CorruptedLogFileException

getNextLogLine

private static WAL.Entry getNextLogLine(WAL.Reader in,
                                        org.apache.hadoop.fs.Path path,
                                        boolean skipErrors)
                                 throws WALSplitter.CorruptedLogFileException,
                                        IOException

Throws:

WALSplitter.CorruptedLogFileException

IOException

createWriter

protected WALProvider.Writer createWriter(org.apache.hadoop.fs.Path logfile)
                                   throws IOException

Create a new WALProvider.Writer for writing log splits.

Returns:

a new Writer instance, caller should close

Throws:

IOException

getReader

protected WAL.Reader getReader(org.apache.hadoop.fs.Path curLogFile,
                               CancelableProgressable reporter)
                        throws IOException

Create a new WAL.Reader for reading logs to split.

Returns:

new Reader instance, caller should close

Throws:

IOException

getNumOpenWriters

private int getNumOpenWriters()

Get current open writers

getMutationsFromWALEntry

public static List<WALSplitter.MutationReplay> getMutationsFromWALEntry(org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos.WALEntry entry,
                                                                        CellScanner cells,
                                                                        Pair<WALKey,WALEdit> logEntry,
                                                                        Durability durability)
                                                                 throws IOException

This function is used to construct mutations from a WALEntry. It also reconstructs WALKey & WALEdit from the passed in WALEntry

Parameters:

entry -

cells -

logEntry - pair of WALKey and WALEdit instance stores WALKey and WALEdit instances extracted from the passed in WALEntry.

Returns:

list of Pair<MutationType, Mutation> to be replayed

Throws:

IOException