org.apache.hadoop.hbase.regionserver.wal
Class HLogSplitter

java.lang.Object
  org.apache.hadoop.hbase.regionserver.wal.HLogSplitter

@InterfaceAudience.Private
public class HLogSplitter
extends Object

This class is responsible for splitting up a bunch of regionserver commit log files that are no longer being written to, into new files, one per region for region to replay on startup. Delete the old log files when finished.

Field Summary

protected org.apache.hadoop.conf.Configuration	conf
protected boolean	distributedLogReplay
protected org.apache.hadoop.fs.FileSystem	fs
protected org.apache.hadoop.fs.Path	oldLogDir
protected org.apache.hadoop.fs.Path	rootDir
protected LastSequenceId	sequenceIdChecker
protected org.apache.hadoop.fs.Path	srcDir
protected AtomicReference<Throwable>	thrown

Constructor Summary

HLogSplitter(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path rootDir, org.apache.hadoop.fs.Path srcDir, org.apache.hadoop.fs.Path oldLogDir, org.apache.hadoop.fs.FileSystem fs, LastSequenceId idChecker)

HLogSplitter(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path rootDir, org.apache.hadoop.fs.Path srcDir, org.apache.hadoop.fs.Path oldLogDir, org.apache.hadoop.fs.FileSystem fs, LastSequenceId idChecker, ZooKeeperWatcher zkw)

Method Summary

static HLogSplitter	createLogSplitter(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path rootDir, org.apache.hadoop.fs.Path srcDir, org.apache.hadoop.fs.Path oldLogDir, org.apache.hadoop.fs.FileSystem fs) Create a new HLogSplitter using the given Configuration and the hbase.hlog.splitter.impl property to derived the instance class to use.
protected HLog.Writer	createWriter(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path logfile, org.apache.hadoop.conf.Configuration conf) Create a new HLog.Writer for writing log splits.
static void	finishSplitLogFile(org.apache.hadoop.fs.Path rootdir, org.apache.hadoop.fs.Path oldLogDir, String logfile, org.apache.hadoop.conf.Configuration conf)
static void	finishSplitLogFile(String logfile, org.apache.hadoop.conf.Configuration conf) Completes the work done by splitLogFile by archiving logs
protected HLog.Reader	getReader(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.FileStatus file, org.apache.hadoop.conf.Configuration conf, boolean skipErrors, CancelableProgressable reporter) Create a new HLog.Reader for reading logs to split.
protected HLog.Reader	getReader(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path curLogFile, org.apache.hadoop.conf.Configuration conf, CancelableProgressable reporter) Create a new HLog.Reader for reading logs to split.
long	getSize()
long	getTime()
List<org.apache.hadoop.fs.Path>	splitLog() Split up a bunch of regionserver commit log files that are no longer being written to, into new files, one per region for region to replay on startup.
List<org.apache.hadoop.fs.Path>	splitLog(CountDownLatch latch) Split up a bunch of regionserver commit log files that are no longer being written to, into new files, one per region for region to replay on startup.
boolean	splitLogFile(org.apache.hadoop.fs.FileStatus logfile, CancelableProgressable reporter)
static boolean	splitLogFile(org.apache.hadoop.fs.Path rootDir, org.apache.hadoop.fs.FileStatus logfile, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.conf.Configuration conf, CancelableProgressable reporter) Splits a HLog file into region's recovered-edits directory
static boolean	splitLogFile(org.apache.hadoop.fs.Path rootDir, org.apache.hadoop.fs.FileStatus logfile, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.conf.Configuration conf, CancelableProgressable reporter, LastSequenceId idChecker, ZooKeeperWatcher zkw) Splits a HLog file into region's recovered-edits directory

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

rootDir

protected final org.apache.hadoop.fs.Path rootDir

srcDir

protected final org.apache.hadoop.fs.Path srcDir

oldLogDir

protected final org.apache.hadoop.fs.Path oldLogDir

fs

protected final org.apache.hadoop.fs.FileSystem fs

conf

protected final org.apache.hadoop.conf.Configuration conf

thrown

protected AtomicReference<Throwable> thrown

sequenceIdChecker

protected final LastSequenceId sequenceIdChecker

distributedLogReplay

protected boolean distributedLogReplay

Constructor Detail

HLogSplitter

public HLogSplitter(org.apache.hadoop.conf.Configuration conf,
                    org.apache.hadoop.fs.Path rootDir,
                    org.apache.hadoop.fs.Path srcDir,
                    org.apache.hadoop.fs.Path oldLogDir,
                    org.apache.hadoop.fs.FileSystem fs,
                    LastSequenceId idChecker)

HLogSplitter

public HLogSplitter(org.apache.hadoop.conf.Configuration conf,
                    org.apache.hadoop.fs.Path rootDir,
                    org.apache.hadoop.fs.Path srcDir,
                    org.apache.hadoop.fs.Path oldLogDir,
                    org.apache.hadoop.fs.FileSystem fs,
                    LastSequenceId idChecker,
                    ZooKeeperWatcher zkw)

Method Detail

createLogSplitter

public static HLogSplitter createLogSplitter(org.apache.hadoop.conf.Configuration conf,
                                             org.apache.hadoop.fs.Path rootDir,
                                             org.apache.hadoop.fs.Path srcDir,
                                             org.apache.hadoop.fs.Path oldLogDir,
                                             org.apache.hadoop.fs.FileSystem fs)

Create a new HLogSplitter using the given Configuration and the hbase.hlog.splitter.impl property to derived the instance class to use. distributedLogReplay won't be enabled by this constructor.

Parameters:

conf -

rootDir - hbase directory

srcDir - logs directory

oldLogDir - directory where processed logs are archived to

fs - FileSystem

Returns:

New HLogSplitter instance

splitLog

public List<org.apache.hadoop.fs.Path> splitLog()
                                         throws IOException

Split up a bunch of regionserver commit log files that are no longer being written to, into new files, one per region for region to replay on startup. Delete the old log files when finished.

Returns:

the list of splits

Throws:

IOException - will throw if corrupted hlogs aren't tolerated

splitLog

public List<org.apache.hadoop.fs.Path> splitLog(CountDownLatch latch)
                                         throws IOException

Split up a bunch of regionserver commit log files that are no longer being written to, into new files, one per region for region to replay on startup. Delete the old log files when finished.

Parameters:

latch -

Returns:

the list of splits

Throws:

IOException - will throw if corrupted hlogs aren't tolerated

getTime

public long getTime()

Returns:

time that this split took

getSize

public long getSize()

Returns:

aggregate size of hlogs that were split

splitLogFile

public static boolean splitLogFile(org.apache.hadoop.fs.Path rootDir,
                                   org.apache.hadoop.fs.FileStatus logfile,
                                   org.apache.hadoop.fs.FileSystem fs,
                                   org.apache.hadoop.conf.Configuration conf,
                                   CancelableProgressable reporter,
                                   LastSequenceId idChecker,
                                   ZooKeeperWatcher zkw)
                            throws IOException

Splits a HLog file into region's recovered-edits directory

If the log file has N regions then N recovered.edits files will be produced.

Parameters:

rootDir -

logfile -

fs -

conf -

reporter -

idChecker -

zkw - ZooKeeperWatcher if it's null, we will back to the old-style log splitting where we dump out recoved.edits files for regions to replay on.

Returns:

false if it is interrupted by the progress-able.

Throws:

IOException

splitLogFile

public static boolean splitLogFile(org.apache.hadoop.fs.Path rootDir,
                                   org.apache.hadoop.fs.FileStatus logfile,
                                   org.apache.hadoop.fs.FileSystem fs,
                                   org.apache.hadoop.conf.Configuration conf,
                                   CancelableProgressable reporter)
                            throws IOException

Splits a HLog file into region's recovered-edits directory

If the log file has N regions then N recovered.edits files will be produced.

Parameters:

rootDir -

logfile -

fs -

conf -

reporter -

Returns:

false if it is interrupted by the progress-able.

Throws:

IOException

splitLogFile

public boolean splitLogFile(org.apache.hadoop.fs.FileStatus logfile,
                            CancelableProgressable reporter)
                     throws IOException

Throws:

IOException

finishSplitLogFile

public static void finishSplitLogFile(String logfile,
                                      org.apache.hadoop.conf.Configuration conf)
                               throws IOException

Completes the work done by splitLogFile by archiving logs

It is invoked by SplitLogManager once it knows that one of the SplitLogWorkers have completed the splitLogFile() part. If the master crashes then this function might get called multiple times.

Parameters:

logfile -

conf -

Throws:

IOException

finishSplitLogFile

public static void finishSplitLogFile(org.apache.hadoop.fs.Path rootdir,
                                      org.apache.hadoop.fs.Path oldLogDir,
                                      String logfile,
                                      org.apache.hadoop.conf.Configuration conf)
                               throws IOException

Throws:

IOException

getReader

protected HLog.Reader getReader(org.apache.hadoop.fs.FileSystem fs,
                                org.apache.hadoop.fs.FileStatus file,
                                org.apache.hadoop.conf.Configuration conf,
                                boolean skipErrors,
                                CancelableProgressable reporter)
                         throws IOException,
                                org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.CorruptedLogFileException

Create a new HLog.Reader for reading logs to split.

Parameters:

fs -

file -

conf -

Returns:

A new Reader instance

Throws:

IOException

CorruptedLogFileException

org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.CorruptedLogFileException

createWriter

protected HLog.Writer createWriter(org.apache.hadoop.fs.FileSystem fs,
                                   org.apache.hadoop.fs.Path logfile,
                                   org.apache.hadoop.conf.Configuration conf)
                            throws IOException

Create a new HLog.Writer for writing log splits.

Throws:

IOException

getReader

protected HLog.Reader getReader(org.apache.hadoop.fs.FileSystem fs,
                                org.apache.hadoop.fs.Path curLogFile,
                                org.apache.hadoop.conf.Configuration conf,
                                CancelableProgressable reporter)
                         throws IOException

Create a new HLog.Reader for reading logs to split.

Throws:

IOException