Package org.apache.hadoop.hbase.regionserver.wal

Class AbstractFSWAL<W extends WALProvider.WriterBase>

java.lang.Object

org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL<W>

All Implemented Interfaces:

Closeable, AutoCloseable, WALFileLengthProvider, WAL

Direct Known Subclasses:

AsyncFSWAL, FSHLog

@Private
public abstract class AbstractFSWAL<W extends WALProvider.WriterBase>
extends Object
implements WAL

Implementation of WAL to go against FileSystem; i.e. keep WALs in HDFS. Only one WAL is ever being written at a time. When a WAL hits a configured maximum size, it is rolled. This is done internal to the implementation.

As data is flushed from the MemStore to other on-disk structures (files sorted by key, hfiles), a WAL becomes obsolete. We can let go of all the log edits/entries for a given HRegion-sequence id. A bunch of work in the below is done keeping account of these region sequence ids -- what is flushed out to hfiles, and what is yet in WAL and in memory only.

It is only practical to delete entire files. Thus, we delete an entire on-disk file F when all of the edits in F have a log-sequence-id that's older (smaller) than the most-recent flush.

To read an WAL, call WALFactory.createStreamReader(FileSystem, Path) for one way read, call WALFactory.createTailingReader(FileSystem, Path, Configuration, long) for replication where we may want to tail the active WAL file.

Failure Semantic

If an exception on append or sync, roll the WAL because the current WAL is now a lame duck; any more appends or syncs will fail also with the same original exception. If we have made successful appends to the WAL and we then are unable to sync them, our current semantic is to return error to the client that the appends failed but also to abort the current context, usually the hosting server. We need to replay the WALs.
TODO: Change this semantic. A roll of WAL may be sufficient as long as we have flagged client that the append failed.
TODO: replication may pick up these last edits though they have been marked as failed append (Need to keep our own file lengths, not rely on HDFS).

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
private static final class	AbstractFSWAL.WALProps

Nested classes/interfaces inherited from interface org.apache.hadoop.hbase.wal.WAL

WAL.Entry

Field Summary

Fields

Modifier and Type	Field	Description
protected final Abortable	abortable
private final int	archiveRetries
private final long	batchSize
protected final long	blocksize	Block size to use writing files.
protected boolean	closed
protected final ExecutorService	closeExecutor
protected final org.apache.hadoop.conf.Configuration	conf	conf object
protected ExecutorService	consumeExecutor
private final Lock	consumeLock
protected final Runnable	consumer
private final AtomicBoolean	consumerScheduled
protected final WALCoprocessorHost	coprocessorHost
protected static final int	DEFAULT_ROLL_ON_SYNC_TIME_MS
protected static final int	DEFAULT_SLOW_SYNC_ROLL_INTERVAL_MS
protected static final int	DEFAULT_SLOW_SYNC_ROLL_THRESHOLD
protected static final int	DEFAULT_SLOW_SYNC_TIME_MS
static final long	DEFAULT_WAL_BATCH_SIZE
static final int	DEFAULT_WAL_SHUTDOWN_WAIT_TIMEOUT_MS
protected static final int	DEFAULT_WAL_SYNC_TIMEOUT_MS
private int	epochAndState
private long	fileLengthAtLastSync
protected final AtomicLong	filenum
protected final org.apache.hadoop.fs.FileSystem	fs	file system instance
protected Supplier<Boolean>	hasConsumerTask
protected long	highestProcessedAppendTxid
private long	highestProcessedAppendTxidAtLastSync
protected final AtomicLong	highestSyncedTxid	Updated to the transaction id of the last successful sync call.
protected long	highestUnsyncedTxid	The highest known outstanding unsync'd WALEdit transaction id.
protected final String	implClassName	The class name of the runtime implementation, used as prefix for logging/tracing.
protected final Map<String,W>	inflightWALClosures	Tracks the logs in the process of being closed.
private long	lastTimeCheckLowReplication
private long	lastTimeCheckSlowSync
protected final List<WALActionsListener>	listeners	Listeners that are called on WAL events.
private static final org.slf4j.Logger	LOG
(package private) final Comparator<org.apache.hadoop.fs.Path>	LOG_NAME_COMPARATOR	WAL Comparator; it compares the timestamp (log filenum), present in the log file name.
private final ExecutorService	logArchiveExecutor
protected final long	logrollsize
private boolean	markerEditOnly
private static final int	MAX_EPOCH
static final String	MAX_LOGS
protected final int	maxLogs
private long	nextLogTooOldNs
protected final AtomicInteger	numEntries
protected final org.apache.hadoop.fs.PathFilter	ourFiles	Matches just those wal files that belong to this wal instance.
protected final String	prefixPathStr	Prefix used when checking for wal membership.
private boolean	readyForRolling
private final Condition	readyForRollingCond
private final org.apache.hadoop.fs.FileSystem	remoteFs
private final org.apache.hadoop.fs.Path	remoteWALDir
static final String	RING_BUFFER_SLOT_COUNT
protected static final String	ROLL_ON_SYNC_TIME_MS
protected final long	rollOnSyncNs	The slow sync will be logged; the very slow sync will cause the WAL to be rolled.
protected final AtomicBoolean	rollRequested
protected final ReentrantLock	rollWriterLock	This lock makes sure only one log roll runs at a time.
private static final Comparator<SyncFuture>	SEQ_COMPARATOR
protected final SequenceIdAccounting	sequenceIdAccounting	Class that does accounting of sequenceids in WAL subsystem.
protected boolean	shouldShutDownConsumeExecutorWhenClose
protected final AtomicBoolean	shutdown
private boolean	skipRemoteWAL
protected static final String	SLOW_SYNC_ROLL_INTERVAL_MS
protected static final String	SLOW_SYNC_ROLL_THRESHOLD
protected static final String	SLOW_SYNC_TIME_MS
protected final int	slowSyncCheckInterval
protected final AtomicInteger	slowSyncCount
protected final long	slowSyncNs	The slow sync will be logged; the very slow sync will cause the WAL to be rolled.
protected final int	slowSyncRollThreshold
private static final long	SURVIVED_TOO_LONG_LOG_INTERVAL_NS	Don't log blocking regions more frequently than this.
private static final int	SURVIVED_TOO_LONG_SEC_DEFAULT
private static final String	SURVIVED_TOO_LONG_SEC_KEY
protected final SyncFutureCache	syncFutureCache	A cache of sync futures reused by threads.
protected final SortedSet<SyncFuture>	syncFutures
protected final AtomicLong	totalLogSize	The total size of wal
protected final Deque<FSWALEntry>	toWriteAppends
protected final Deque<FSWALEntry>	unackedAppends
protected final boolean	useHsync
private final com.lmax.disruptor.RingBuffer<RingBufferTruck>	waitingConsumePayloads
private final com.lmax.disruptor.Sequence	waitingConsumePayloadsGatingSequence
private int	waitOnShutdownInSeconds
private String	waitOnShutdownInSecondsConfigKey
static final boolean	WAL_AVOID_LOCAL_WRITES_DEFAULT
static final String	WAL_AVOID_LOCAL_WRITES_KEY
static final String	WAL_BATCH_SIZE
static final String	WAL_ROLL_MULTIPLIER
static final String	WAL_SHUTDOWN_WAIT_TIMEOUT_MS
static final String	WAL_SYNC_TIMEOUT_MS
protected final org.apache.hadoop.fs.Path	walArchiveDir	dir path where old logs are kept.
protected final org.apache.hadoop.fs.Path	walDir	WAL directory, where all WAL files would be placed.
protected final ConcurrentNavigableMap<org.apache.hadoop.fs.Path,AbstractFSWAL.WALProps>	walFile2Props	Map of WAL log file to properties.
protected final String	walFilePrefix	Prefix of a WAL file, usually the region server name it is hosted on.
protected final String	walFileSuffix	Suffix included on generated wal file names
protected final long	walShutdownTimeout
private final long	walSyncTimeoutNs
private final long	walTooOldNs
(package private) W	writer	Current log file.

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	AbstractFSWAL(org.apache.hadoop.fs.FileSystem fs, Abortable abortable, org.apache.hadoop.fs.Path rootDir, String logDir, String archiveDir, org.apache.hadoop.conf.Configuration conf, List<WALActionsListener> listeners, boolean failIfWALExists, String prefix, String suffix, org.apache.hadoop.fs.FileSystem remoteFs, org.apache.hadoop.fs.Path remoteWALDir)

Method Summary

Modifier and Type	Method	Description
void	abortCacheFlush(byte[] encodedRegionName)	Abort a cache flush.
protected long	append(RegionInfo hri, WALKeyImpl key, WALEdit edits, boolean inMemstore)	Append a set of edits to the WAL.
private void	appendAndSync()
long	appendData(RegionInfo info, WALKeyImpl key, WALEdit edits)	Append a set of data edits to the WAL.
protected final boolean	appendEntry(W writer, FSWALEntry entry)
long	appendMarker(RegionInfo info, WALKeyImpl key, WALEdit edits)	Append an operational 'meta' event marker edit to the WAL.
protected void	archive(Pair<org.apache.hadoop.fs.Path,Long> log)
protected void	archiveLogFile(org.apache.hadoop.fs.Path p)
protected void	atHeadOfRingBufferEventHandlerAppend()	Exposed for testing only.
protected final void	blockOnSync(SyncFuture syncFuture)
private int	calculateMaxLogFiles(org.apache.hadoop.conf.Configuration conf, long logRollSize)
void	checkLogLowReplication(long checkInterval)
protected void	checkSlowSyncCount()
private void	cleanOldLogs()	Archive old logs.
void	close()	Caller no longer needs any edits from this WAL.
protected final void	closeWriter(W writer, org.apache.hadoop.fs.Path path)
void	completeCacheFlush(byte[] encodedRegionName, long maxFlushedSeqId)	Complete the cache flush.
protected org.apache.hadoop.fs.Path	computeFilename(long filenum)	This is a convenience method that computes a new filename with a given file-number.
private void	consume()
private IOException	convertInterruptedExceptionToIOException(InterruptedException ie)
private W	createCombinedWriter(W localWriter, org.apache.hadoop.fs.Path localPath)
protected abstract W	createCombinedWriter(W localWriter, W remoteWriter)
protected final void	createSingleThreadPoolConsumeExecutor(String walType, org.apache.hadoop.fs.Path rootDir, String prefix)
private io.opentelemetry.api.trace.Span	createSpan(String name)
protected abstract W	createWriterInstance(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path)
protected abstract void	doAppend(W writer, FSWALEntry entry)
protected abstract boolean	doCheckLogLowReplication()
protected boolean	doCheckSlowSync()	Returns true if we exceeded the slow sync roll threshold over the last check interval
protected void	doCleanUpResources()
protected void	doReplaceWriter(org.apache.hadoop.fs.Path oldPath, org.apache.hadoop.fs.Path newPath, W nextWriter)	Notice that you need to clear the rollRequested flag in this method, as the new writer will begin to work before returning from this method.
protected void	doShutdown()
protected void	doSync(boolean forceSync)
protected void	doSync(long txid, boolean forceSync)
protected abstract CompletableFuture<Long>	doWriterSync(W writer, boolean shouldUseHsync, long txidWhenSyn)
private void	drainNonMarkerEditsAndFailSyncs()
private static IOException	ensureIOException(Throwable t)
private static int	epoch(int epochAndState)
(package private) Map<byte[],List<byte[]>>	findRegionsToForceFlush()	If the number of un-archived WAL files ('live' WALs) is greater than maximum allowed, check the first (oldest) WAL, and return those regions which should be flushed so that it can be let-go/'archived'.
private int	finishSync()
private int	finishSyncLowerThanTxid(long txid)
WALCoprocessorHost	getCoprocessorHost()	Returns Coprocessor host.
org.apache.hadoop.fs.Path	getCurrentFileName()	This is a convenience method that computes a new filename with a given using the current WAL file-number
long	getEarliestMemStoreSeqNum(byte[] encodedRegionName, byte[] familyName)	Gets the earliest unflushed sequence id in the memstore for the store.
long	getFilenum()
protected long	getFileNumFromFileName(org.apache.hadoop.fs.Path fileName)	A log file has a creation timestamp (in ms) in its file name (filenum.
(package private) org.apache.hadoop.fs.FileStatus[]	getFiles()	Get the backing files associated with this WAL.
int	getInflightWALCloseCount()	Returns number of WALs currently in the process of closing.
private static long	getLastTxid(Deque<FSWALEntry> queue)
long	getLogFileSize()	Returns the size of log files in use
OptionalLong	getLogFileSizeIfBeingWritten(org.apache.hadoop.fs.Path path)	if the given path is being written currently, then return its length.
(package private) abstract int	getLogReplication()	This method gets the datanode replication count for the current WAL.
private org.apache.hadoop.fs.Path	getNewPath()	retrieve the next path to use for writing.
int	getNumLogFiles()	Returns the number of log files in use
int	getNumRolledLogFiles()	Returns the number of rolled log files
org.apache.hadoop.fs.Path	getOldPath()
(package private) abstract org.apache.hadoop.hdfs.protocol.DatanodeInfo[]	getPipeline()	This method gets the pipeline for the current WAL.
protected final int	getPreallocatedEventCount()
SequenceIdAccounting	getSequenceIdAccounting()
protected long	getSyncedTxid(long processedTxid, long completableFutureResult)	This method is to adapt FSHLog and AsyncFSWAL.
protected final SyncFuture	getSyncFuture(long sequence, boolean forceSync)
(package private) long	getUnflushedEntriesCount()
static org.apache.hadoop.fs.Path	getWALArchivePath(org.apache.hadoop.fs.Path archiveDir, org.apache.hadoop.fs.Path p)
(package private) W	getWriter()
void	init()	Used to initialize the WAL.
private boolean	isHsync(long beginTxid, long endTxid)
protected boolean	isLogRollRequested()
(package private) boolean	isUnflushedEntries()
protected boolean	isWriterBroken()
protected final void	logRollAndSetupWalProps(org.apache.hadoop.fs.Path oldPath, org.apache.hadoop.fs.Path newPath, long oldFileLen)
static void	main(String[] args)	Pass one or more log file names, and it will either dump out a text version on stdout or split the specified log files.
private void	markClosedAndClean(org.apache.hadoop.fs.Path path)	Mark this WAL file as closed and call cleanOldLogs to see if we can archive this file.
protected void	markFutureDoneAndOffer(SyncFuture future, long txid, Throwable t)	Helper that marks the future as DONE and offers it back to the cache.
private void	onAppendEntryFailed(IOException exception)
private void	onException(long epochWhenSync, Throwable error)
protected abstract void	onWriterReplaced(W nextWriter)
private long	postAppend(WAL.Entry e, long elapsedTime)
protected final void	postSync(long timeInNanos, int handlerSyncs)
private void	recoverLease(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path p, org.apache.hadoop.conf.Configuration conf)
void	registerWALActionsListener(WALActionsListener listener)	Registers WALActionsListener
(package private) org.apache.hadoop.fs.Path	replaceWriter(org.apache.hadoop.fs.Path oldPath, org.apache.hadoop.fs.Path newPath, W nextWriter)	Cleans up current writer closing it and then puts in place the passed in nextWriter.
void	requestLogRoll()
protected final void	requestLogRoll(WALActionsListener.RollRequestReason reason)
Map<byte[],List<byte[]>>	rollWriter()	Roll the log writer.
Map<byte[],List<byte[]>>	rollWriter(boolean force)	Roll the log writer.
private Map<byte[],List<byte[]>>	rollWriterInternal(boolean force)
protected final void	setWaitOnShutdownInSeconds(int waitOnShutdownInSeconds, String waitOnShutdownInSecondsConfigKey)
private boolean	shouldScheduleConsumer()
void	shutdown()	Stop accepting new writes.
void	skipRemoteWAL(boolean markerEditOnly)	Tell the WAL that when creating new writer you can skip creating the remote writer.
private static void	split(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path p)
protected final long	stampSequenceIdAndPublishToRingBuffer(RegionInfo hri, WALKeyImpl key, WALEdit edits, boolean inMemstore, com.lmax.disruptor.RingBuffer<RingBufferTruck> ringBuffer)
Long	startCacheFlush(byte[] encodedRegionName, Map<byte[],Long> familyToSeq)
Long	startCacheFlush(byte[] encodedRegionName, Set<byte[]> families)	WAL keeps track of the sequence numbers that are as yet not flushed im memstores in order to be able to do accounting to figure which WALs can be let go.
final void	sync()	Sync what we have in the WAL.
final void	sync(boolean forceSync)
final void	sync(long txid)	Sync the WAL if the txId was not already sync'd.
final void	sync(long txid, boolean forceSync)
private void	sync(W writer)
private void	syncCompleted(long epochWhenSync, W writer, long processedTxid, long startTimeNs)
private void	syncFailed(long epochWhenSync, Throwable error)
private void	tellListenersAboutPostLogRoll(org.apache.hadoop.fs.Path oldPath, org.apache.hadoop.fs.Path newPath)	Tell listeners about post log roll.
private void	tellListenersAboutPreLogRoll(org.apache.hadoop.fs.Path oldPath, org.apache.hadoop.fs.Path newPath)	Tell listeners about pre log roll.
String	toString()	Human readable identifying information about the state of this WAL.
private boolean	trySetReadyForRolling()
boolean	unregisterWALActionsListener(WALActionsListener listener)	Unregisters WALActionsListener
void	updateStore(byte[] encodedRegionName, byte[] familyName, Long sequenceid, boolean onlyIfGreater)	updates the sequence number of a specific store.
private static void	usage()
protected final void	waitForSafePoint()
private static boolean	waitingRoll(int epochAndState)
private static boolean	writerBroken(int epochAndState)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Details

LOG

private static final org.slf4j.Logger LOG

SEQ_COMPARATOR

private static final Comparator<SyncFuture> SEQ_COMPARATOR

SURVIVED_TOO_LONG_SEC_KEY

private static final String SURVIVED_TOO_LONG_SEC_KEY

See Also:

• Constant Field Values

SURVIVED_TOO_LONG_SEC_DEFAULT

private static final int SURVIVED_TOO_LONG_SEC_DEFAULT

See Also:

• Constant Field Values

SURVIVED_TOO_LONG_LOG_INTERVAL_NS

private static final long SURVIVED_TOO_LONG_LOG_INTERVAL_NS

Don't log blocking regions more frequently than this.

SLOW_SYNC_TIME_MS

protected static final String SLOW_SYNC_TIME_MS

See Also:

• Constant Field Values

DEFAULT_SLOW_SYNC_TIME_MS

protected static final int DEFAULT_SLOW_SYNC_TIME_MS

See Also:

• Constant Field Values

ROLL_ON_SYNC_TIME_MS

protected static final String ROLL_ON_SYNC_TIME_MS

See Also:

• Constant Field Values

DEFAULT_ROLL_ON_SYNC_TIME_MS

protected static final int DEFAULT_ROLL_ON_SYNC_TIME_MS

See Also:

• Constant Field Values

SLOW_SYNC_ROLL_THRESHOLD

protected static final String SLOW_SYNC_ROLL_THRESHOLD

See Also:

• Constant Field Values

DEFAULT_SLOW_SYNC_ROLL_THRESHOLD

protected static final int DEFAULT_SLOW_SYNC_ROLL_THRESHOLD

See Also:

• Constant Field Values

SLOW_SYNC_ROLL_INTERVAL_MS

protected static final String SLOW_SYNC_ROLL_INTERVAL_MS

See Also:

• Constant Field Values

DEFAULT_SLOW_SYNC_ROLL_INTERVAL_MS

protected static final int DEFAULT_SLOW_SYNC_ROLL_INTERVAL_MS

See Also:

• Constant Field Values

WAL_SYNC_TIMEOUT_MS

public static final String WAL_SYNC_TIMEOUT_MS

See Also:

• Constant Field Values

DEFAULT_WAL_SYNC_TIMEOUT_MS

protected static final int DEFAULT_WAL_SYNC_TIMEOUT_MS

See Also:

• Constant Field Values

WAL_ROLL_MULTIPLIER

public static final String WAL_ROLL_MULTIPLIER

See Also:

• Constant Field Values

MAX_LOGS

public static final String MAX_LOGS

See Also:

• Constant Field Values

RING_BUFFER_SLOT_COUNT

public static final String RING_BUFFER_SLOT_COUNT

See Also:

• Constant Field Values

WAL_SHUTDOWN_WAIT_TIMEOUT_MS

public static final String WAL_SHUTDOWN_WAIT_TIMEOUT_MS

See Also:

• Constant Field Values

DEFAULT_WAL_SHUTDOWN_WAIT_TIMEOUT_MS

public static final int DEFAULT_WAL_SHUTDOWN_WAIT_TIMEOUT_MS

See Also:

• Constant Field Values

WAL_BATCH_SIZE

public static final String WAL_BATCH_SIZE

See Also:

• Constant Field Values

DEFAULT_WAL_BATCH_SIZE

public static final long DEFAULT_WAL_BATCH_SIZE

See Also:

• Constant Field Values

WAL_AVOID_LOCAL_WRITES_KEY

public static final String WAL_AVOID_LOCAL_WRITES_KEY

See Also:

• Constant Field Values

WAL_AVOID_LOCAL_WRITES_DEFAULT

public static final boolean WAL_AVOID_LOCAL_WRITES_DEFAULT

See Also:

• Constant Field Values

fs

protected final org.apache.hadoop.fs.FileSystem fs

file system instance

walDir

protected final org.apache.hadoop.fs.Path walDir

WAL directory, where all WAL files would be placed.

remoteFs

private final org.apache.hadoop.fs.FileSystem remoteFs

remoteWALDir

private final org.apache.hadoop.fs.Path remoteWALDir

walArchiveDir

protected final org.apache.hadoop.fs.Path walArchiveDir

dir path where old logs are kept.

ourFiles

protected final org.apache.hadoop.fs.PathFilter ourFiles

Matches just those wal files that belong to this wal instance.

walFilePrefix

protected final String walFilePrefix

Prefix of a WAL file, usually the region server name it is hosted on.

walFileSuffix

protected final String walFileSuffix

Suffix included on generated wal file names

prefixPathStr

protected final String prefixPathStr

Prefix used when checking for wal membership.

coprocessorHost

protected final WALCoprocessorHost coprocessorHost

conf

protected final org.apache.hadoop.conf.Configuration conf

conf object

abortable

protected final Abortable abortable

listeners

protected final List<WALActionsListener> listeners

Listeners that are called on WAL events.

inflightWALClosures

protected final Map<String,W extends WALProvider.WriterBase> inflightWALClosures

Tracks the logs in the process of being closed.

sequenceIdAccounting

protected final SequenceIdAccounting sequenceIdAccounting

Class that does accounting of sequenceids in WAL subsystem. Holds oldest outstanding sequence id as yet not flushed as well as the most recent edit sequence id appended to the WAL. Has facility for answering questions such as "Is it safe to GC a WAL?".

slowSyncNs

protected final long slowSyncNs

The slow sync will be logged; the very slow sync will cause the WAL to be rolled.

rollOnSyncNs

protected final long rollOnSyncNs

The slow sync will be logged; the very slow sync will cause the WAL to be rolled.

slowSyncRollThreshold

protected final int slowSyncRollThreshold

slowSyncCheckInterval

protected final int slowSyncCheckInterval

slowSyncCount

protected final AtomicInteger slowSyncCount

walSyncTimeoutNs

private final long walSyncTimeoutNs

walTooOldNs

private final long walTooOldNs

logrollsize

protected final long logrollsize

blocksize

protected final long blocksize

Block size to use writing files.

maxLogs

protected final int maxLogs

useHsync

protected final boolean useHsync

rollWriterLock

protected final ReentrantLock rollWriterLock

This lock makes sure only one log roll runs at a time. Should not be taken while any other lock is held. We don't just use synchronized because that results in bogus and tedious findbugs warning when it thinks synchronized controls writer thread safety. It is held when we are actually rolling the log. It is checked when we are looking to see if we should roll the log or not.

filenum

protected final AtomicLong filenum

numEntries

protected final AtomicInteger numEntries

highestUnsyncedTxid

protected volatile long highestUnsyncedTxid

The highest known outstanding unsync'd WALEdit transaction id. Usually, we use a queue to pass WALEdit to background consumer thread, and the transaction id is the sequence number of the corresponding entry in queue.

highestSyncedTxid

protected final AtomicLong highestSyncedTxid

Updated to the transaction id of the last successful sync call. This can be less than highestUnsyncedTxid for case where we have an append where a sync has not yet come in for it.

totalLogSize

protected final AtomicLong totalLogSize

The total size of wal

writer

volatile W extends WALProvider.WriterBase writer

Current log file.

lastTimeCheckLowReplication

private volatile long lastTimeCheckLowReplication

lastTimeCheckSlowSync

private volatile long lastTimeCheckSlowSync

closed

protected volatile boolean closed

shutdown

protected final AtomicBoolean shutdown

walShutdownTimeout

protected final long walShutdownTimeout

nextLogTooOldNs

private long nextLogTooOldNs

LOG_NAME_COMPARATOR

final Comparator<org.apache.hadoop.fs.Path> LOG_NAME_COMPARATOR

WAL Comparator; it compares the timestamp (log filenum), present in the log file name. Throws an IllegalArgumentException if used to compare paths from different wals.

walFile2Props

protected final ConcurrentNavigableMap<org.apache.hadoop.fs.Path,AbstractFSWAL.WALProps> walFile2Props

Map of WAL log file to properties. The map is sorted by the log file creation timestamp (contained in the log file name).

syncFutureCache

protected final SyncFutureCache syncFutureCache

A cache of sync futures reused by threads.

implClassName

protected final String implClassName

The class name of the runtime implementation, used as prefix for logging/tracing.

Performance testing shows getClass().getSimpleName() might be a bottleneck so we store it here, refer to HBASE-17676 for more details

rollRequested

protected final AtomicBoolean rollRequested

closeExecutor

protected final ExecutorService closeExecutor

logArchiveExecutor

private final ExecutorService logArchiveExecutor

archiveRetries

private final int archiveRetries

consumeExecutor

protected ExecutorService consumeExecutor

consumeLock

private final Lock consumeLock

consumer

protected final Runnable consumer

hasConsumerTask

protected Supplier<Boolean> hasConsumerTask

MAX_EPOCH

private static final int MAX_EPOCH

See Also:

• Constant Field Values

epochAndState

private volatile int epochAndState

readyForRolling

private boolean readyForRolling

readyForRollingCond

private final Condition readyForRollingCond

waitingConsumePayloads

private final com.lmax.disruptor.RingBuffer<RingBufferTruck> waitingConsumePayloads

waitingConsumePayloadsGatingSequence

private final com.lmax.disruptor.Sequence waitingConsumePayloadsGatingSequence

consumerScheduled

private final AtomicBoolean consumerScheduled

batchSize

private final long batchSize

toWriteAppends

protected final Deque<FSWALEntry> toWriteAppends

unackedAppends

protected final Deque<FSWALEntry> unackedAppends

syncFutures

protected final SortedSet<SyncFuture> syncFutures

highestProcessedAppendTxid

protected long highestProcessedAppendTxid

fileLengthAtLastSync

private long fileLengthAtLastSync

highestProcessedAppendTxidAtLastSync

private long highestProcessedAppendTxidAtLastSync

waitOnShutdownInSeconds

private int waitOnShutdownInSeconds

waitOnShutdownInSecondsConfigKey

private String waitOnShutdownInSecondsConfigKey

shouldShutDownConsumeExecutorWhenClose

protected boolean shouldShutDownConsumeExecutorWhenClose

skipRemoteWAL

private volatile boolean skipRemoteWAL

markerEditOnly

private volatile boolean markerEditOnly

Constructor Details

AbstractFSWAL

protected AbstractFSWAL(org.apache.hadoop.fs.FileSystem fs,
 Abortable abortable,
 org.apache.hadoop.fs.Path rootDir,
 String logDir,
 String archiveDir,
 org.apache.hadoop.conf.Configuration conf,
 List<WALActionsListener> listeners,
 boolean failIfWALExists,
 String prefix,
 String suffix,
 org.apache.hadoop.fs.FileSystem remoteFs,
 org.apache.hadoop.fs.Path remoteWALDir)
                 throws FailedLogCloseException,
IOException

Throws:

FailedLogCloseException

IOException

Method Details

getFilenum

public long getFilenum()

getFileNumFromFileName

protected long getFileNumFromFileName(org.apache.hadoop.fs.Path fileName)

A log file has a creation timestamp (in ms) in its file name (filenum. This helper method returns the creation timestamp from a given log file. It extracts the timestamp assuming the filename is created with the computeFilename(long filenum) method.

Returns:

timestamp, as in the log file name.

calculateMaxLogFiles

private int calculateMaxLogFiles(org.apache.hadoop.conf.Configuration conf,
 long logRollSize)

getPreallocatedEventCount

protected final int getPreallocatedEventCount()

setWaitOnShutdownInSeconds

protected final void setWaitOnShutdownInSeconds(int waitOnShutdownInSeconds,
 String waitOnShutdownInSecondsConfigKey)

createSingleThreadPoolConsumeExecutor

protected final void createSingleThreadPoolConsumeExecutor(String walType,
 org.apache.hadoop.fs.Path rootDir,
 String prefix)

init

public void init()
          throws IOException

Used to initialize the WAL. Usually just call rollWriter to create the first log writer.

Specified by:

init in interface WAL

Throws:

IOException

registerWALActionsListener

public void registerWALActionsListener(WALActionsListener listener)

Description copied from interface: WAL

Registers WALActionsListener

Specified by:

registerWALActionsListener in interface WAL

unregisterWALActionsListener

public boolean unregisterWALActionsListener(WALActionsListener listener)

Description copied from interface: WAL

Unregisters WALActionsListener

Specified by:

unregisterWALActionsListener in interface WAL

getCoprocessorHost

public WALCoprocessorHost getCoprocessorHost()

Description copied from interface: WAL

Returns Coprocessor host.

Specified by:

getCoprocessorHost in interface WAL

startCacheFlush

public Long startCacheFlush(byte[] encodedRegionName,
 Set<byte[]> families)

Description copied from interface: WAL

WAL keeps track of the sequence numbers that are as yet not flushed im memstores in order to be able to do accounting to figure which WALs can be let go. This method tells WAL that some region is about to flush. The flush can be the whole region or for a column family of the region only.

Currently, it is expected that the update lock is held for the region; i.e. no concurrent appends while we set up cache flush.

Specified by:

startCacheFlush in interface WAL

families - Families to flush. May be a subset of all families in the region.

Returns:

Returns HConstants.NO_SEQNUM if we are flushing the whole region OR if we are flushing a subset of all families but there are no edits in those families not being flushed; in other words, this is effectively same as a flush of all of the region though we were passed a subset of regions. Otherwise, it returns the sequence id of the oldest/lowest outstanding edit.

See Also:

• WAL.completeCacheFlush(byte[], long)
• WAL.abortCacheFlush(byte[])

startCacheFlush

public Long startCacheFlush(byte[] encodedRegionName,
 Map<byte[],Long> familyToSeq)

Specified by:

startCacheFlush in interface WAL

completeCacheFlush

public void completeCacheFlush(byte[] encodedRegionName,
 long maxFlushedSeqId)

Description copied from interface: WAL

Complete the cache flush.

Specified by:

completeCacheFlush in interface WAL

Parameters:

encodedRegionName - Encoded region name.

maxFlushedSeqId - The maxFlushedSeqId for this flush. There is no edit in memory that is less that this sequence id.

See Also:

• WAL.startCacheFlush(byte[], Set)
• WAL.abortCacheFlush(byte[])

abortCacheFlush

public void abortCacheFlush(byte[] encodedRegionName)

Description copied from interface: WAL

Abort a cache flush. Call if the flush fails. Note that the only recovery for an aborted flush currently is a restart of the regionserver so the snapshot content dropped by the failure gets restored to the memstore.

Specified by:

abortCacheFlush in interface WAL

Parameters:

encodedRegionName - Encoded region name.

getEarliestMemStoreSeqNum

public long getEarliestMemStoreSeqNum(byte[] encodedRegionName,
 byte[] familyName)

Description copied from interface: WAL

Gets the earliest unflushed sequence id in the memstore for the store.

Specified by:

getEarliestMemStoreSeqNum in interface WAL

Parameters:

encodedRegionName - The region to get the number for.

familyName - The family to get the number for.

Returns:

The earliest/lowest/oldest sequence id if present, HConstants.NO_SEQNUM if absent.

rollWriter

public Map<byte[],List<byte[]>> rollWriter()
                                    throws FailedLogCloseException,
IOException

Description copied from interface: WAL

Roll the log writer. That is, start writing log messages to a new file.

The implementation is synchronized in order to make sure there's one rollWriter running at any given time.

Specified by:

rollWriter in interface WAL

Returns:

If lots of logs, flush the stores of returned regions so next time through we can clean logs. Returns null if nothing to flush. Names are actual region names as returned by RegionInfo.getEncodedName()

Throws:

FailedLogCloseException

IOException

sync

public final void sync()
                throws IOException

Description copied from interface: WAL

Sync what we have in the WAL.

Specified by:

sync in interface WAL

Throws:

IOException

sync

public final void sync(long txid)
                throws IOException

Description copied from interface: WAL

Sync the WAL if the txId was not already sync'd.

Specified by:

sync in interface WAL

Parameters:

txid - Transaction id to sync to.

Throws:

IOException

sync

public final void sync(boolean forceSync)
                throws IOException

Specified by:

sync in interface WAL

Parameters:

forceSync - Flag to force sync rather than flushing to the buffer. Example - Hadoop hflush vs hsync.

Throws:

IOException

sync

public final void sync(long txid,
 boolean forceSync)
                throws IOException

Specified by:

sync in interface WAL

Parameters:

txid - Transaction id to sync to.

forceSync - Flag to force sync rather than flushing to the buffer. Example - Hadoop hflush vs hsync.

Throws:

IOException

getSequenceIdAccounting

public SequenceIdAccounting getSequenceIdAccounting()

computeFilename

protected org.apache.hadoop.fs.Path computeFilename(long filenum)

This is a convenience method that computes a new filename with a given file-number.

Parameters:

filenum - to use

getCurrentFileName

public org.apache.hadoop.fs.Path getCurrentFileName()

This is a convenience method that computes a new filename with a given using the current WAL file-number

getNewPath

private org.apache.hadoop.fs.Path getNewPath()
                                      throws IOException

retrieve the next path to use for writing. Increments the internal filenum.

Throws:

IOException

getOldPath

public org.apache.hadoop.fs.Path getOldPath()

tellListenersAboutPreLogRoll

private void tellListenersAboutPreLogRoll(org.apache.hadoop.fs.Path oldPath,
 org.apache.hadoop.fs.Path newPath)
                                   throws IOException

Tell listeners about pre log roll.

Throws:

IOException

tellListenersAboutPostLogRoll

private void tellListenersAboutPostLogRoll(org.apache.hadoop.fs.Path oldPath,
 org.apache.hadoop.fs.Path newPath)
                                    throws IOException

Tell listeners about post log roll.

Throws:

IOException

getNumRolledLogFiles

public int getNumRolledLogFiles()

Returns the number of rolled log files

getNumLogFiles

public int getNumLogFiles()

Returns the number of log files in use

findRegionsToForceFlush

Map<byte[],List<byte[]>> findRegionsToForceFlush()
                                          throws IOException

If the number of un-archived WAL files ('live' WALs) is greater than maximum allowed, check the first (oldest) WAL, and return those regions which should be flushed so that it can be let-go/'archived'.

Returns:

stores of regions (encodedRegionNames) to flush in order to archive the oldest WAL file

Throws:

IOException

markClosedAndClean

private void markClosedAndClean(org.apache.hadoop.fs.Path path)

Mark this WAL file as closed and call cleanOldLogs to see if we can archive this file.

cleanOldLogs

private void cleanOldLogs()

Archive old logs. A WAL is eligible for archiving if all its WALEdits have been flushed.

Use synchronized because we may call this method in different threads, normally when replacing writer, and since now close writer may be asynchronous, we will also call this method in the closeExecutor, right after we actually close a WAL writer.

archive

protected void archive(Pair<org.apache.hadoop.fs.Path,Long> log)

getWALArchivePath

public static org.apache.hadoop.fs.Path getWALArchivePath(org.apache.hadoop.fs.Path archiveDir,
 org.apache.hadoop.fs.Path p)

archiveLogFile

protected void archiveLogFile(org.apache.hadoop.fs.Path p)
                       throws IOException

Throws:

IOException

logRollAndSetupWalProps

protected final void logRollAndSetupWalProps(org.apache.hadoop.fs.Path oldPath,
 org.apache.hadoop.fs.Path newPath,
 long oldFileLen)

createSpan

private io.opentelemetry.api.trace.Span createSpan(String name)

replaceWriter

org.apache.hadoop.fs.Path replaceWriter(org.apache.hadoop.fs.Path oldPath,
 org.apache.hadoop.fs.Path newPath,
 W nextWriter)
                                 throws IOException

Cleans up current writer closing it and then puts in place the passed in nextWriter.

• In the case of creating a new WAL, oldPath will be null.
• In the case of rolling over from one file to the next, none of the parameters will be null.
• In the case of closing out this FSHLog with no further use newPath and nextWriter will be null.

Parameters:

oldPath - may be null

newPath - may be null

nextWriter - may be null

Returns:

the passed in newPath

Throws:

IOException - if there is a problem flushing or closing the underlying FS

blockOnSync

protected final void blockOnSync(SyncFuture syncFuture)
                          throws IOException

Throws:

IOException

ensureIOException

private static IOException ensureIOException(Throwable t)

convertInterruptedExceptionToIOException

private IOException convertInterruptedExceptionToIOException(InterruptedException ie)

createCombinedWriter

private W createCombinedWriter(W localWriter,
 org.apache.hadoop.fs.Path localPath)
                        throws IOException,
CommonFSUtils.StreamLacksCapabilityException

Throws:

IOException

CommonFSUtils.StreamLacksCapabilityException

rollWriterInternal

private Map<byte[],List<byte[]>> rollWriterInternal(boolean force)
                                             throws IOException

Throws:

IOException

rollWriter

public Map<byte[],List<byte[]>> rollWriter(boolean force)
                                    throws IOException

Description copied from interface: WAL

Roll the log writer. That is, start writing log messages to a new file.

The implementation is synchronized in order to make sure there's one rollWriter running at any given time. If true, force creation of a new writer even if no entries have been written to the current writer

Specified by:

rollWriter in interface WAL

Returns:

If lots of logs, flush the stores of returned regions so next time through we can clean logs. Returns null if nothing to flush. Names are actual region names as returned by RegionInfo.getEncodedName()

Throws:

IOException

getLogFileSize

public long getLogFileSize()

Returns the size of log files in use

requestLogRoll

public void requestLogRoll()

getFiles

org.apache.hadoop.fs.FileStatus[] getFiles()
                                    throws IOException

Get the backing files associated with this WAL.

Returns:

may be null if there are no files.

Throws:

IOException

shutdown

public void shutdown()
              throws IOException

Description copied from interface: WAL

Stop accepting new writes. If we have unsynced writes still in buffer, sync them. Extant edits are left in place in backing storage to be replayed later.

Specified by:

shutdown in interface WAL

Throws:

IOException

close

public void close()
           throws IOException

Description copied from interface: WAL

Caller no longer needs any edits from this WAL. Implementers are free to reclaim underlying resources after this call; i.e. filesystem based WALs can archive or delete files.

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable

Specified by:

close in interface WAL

Throws:

IOException

getInflightWALCloseCount

public int getInflightWALCloseCount()

Returns number of WALs currently in the process of closing.

updateStore

public void updateStore(byte[] encodedRegionName,
 byte[] familyName,
 Long sequenceid,
 boolean onlyIfGreater)

updates the sequence number of a specific store. depending on the flag: replaces current seq number if the given seq id is bigger, or even if it is lower than existing one

Specified by:

updateStore in interface WAL

getSyncFuture

protected final SyncFuture getSyncFuture(long sequence,
 boolean forceSync)

isLogRollRequested

protected boolean isLogRollRequested()

requestLogRoll

protected final void requestLogRoll(WALActionsListener.RollRequestReason reason)

getUnflushedEntriesCount

long getUnflushedEntriesCount()

isUnflushedEntries

boolean isUnflushedEntries()

atHeadOfRingBufferEventHandlerAppend

protected void atHeadOfRingBufferEventHandlerAppend()

Exposed for testing only. Use to tricks like halt the ring buffer appending.

appendEntry

protected final boolean appendEntry(W writer,
 FSWALEntry entry)
                             throws IOException

Throws:

IOException

postAppend

private long postAppend(WAL.Entry e,
 long elapsedTime)
                 throws IOException

Throws:

IOException

postSync

protected final void postSync(long timeInNanos,
 int handlerSyncs)

stampSequenceIdAndPublishToRingBuffer

protected final long stampSequenceIdAndPublishToRingBuffer(RegionInfo hri,
 WALKeyImpl key,
 WALEdit edits,
 boolean inMemstore,
 com.lmax.disruptor.RingBuffer<RingBufferTruck> ringBuffer)
                                                    throws IOException

Throws:

IOException

toString

public String toString()

Description copied from interface: WAL

Human readable identifying information about the state of this WAL. Implementors are encouraged to include information appropriate for debugging. Consumers are advised not to rely on the details of the returned String; it does not have a defined structure.

Specified by:

toString in interface WAL

Overrides:

toString in class Object

getLogFileSizeIfBeingWritten

public OptionalLong getLogFileSizeIfBeingWritten(org.apache.hadoop.fs.Path path)

if the given path is being written currently, then return its length.

This is used by replication to prevent replicating unacked log entries. See https://issues.apache.org/jira/browse/HBASE-14004 for more details.

Specified by:

getLogFileSizeIfBeingWritten in interface WALFileLengthProvider

appendData

public long appendData(RegionInfo info,
 WALKeyImpl key,
 WALEdit edits)
                throws IOException

Description copied from interface: WAL

Append a set of data edits to the WAL. 'Data' here means that the content in the edits will also have transitioned through the memstore.

The WAL is not flushed/sync'd after this transaction completes BUT on return this edit must have its region edit/sequence id assigned else it messes up our unification of mvcc and sequenceid. On return key will have the region edit/sequence id filled in.

Specified by:

appendData in interface WAL

Parameters:

info - the regioninfo associated with append

key - Modified by this call; we add to it this edits region edit/sequence id.

edits - Edits to append. MAY CONTAIN NO EDITS for case where we want to get an edit sequence id that is after all currently appended edits.

Returns:

Returns a 'transaction id' and key will have the region edit/sequence id in it.

Throws:

IOException

See Also:

• WAL.appendMarker(RegionInfo, WALKeyImpl, WALEdit)

appendMarker

public long appendMarker(RegionInfo info,
 WALKeyImpl key,
 WALEdit edits)
                  throws IOException

Description copied from interface: WAL

Append an operational 'meta' event marker edit to the WAL. A marker meta edit could be a FlushDescriptor, a compaction marker, or a region event marker; e.g. region open or region close. The difference between a 'marker' append and a 'data' append as in WAL.appendData(RegionInfo, WALKeyImpl, WALEdit)is that a marker will not have transitioned through the memstore.

The WAL is not flushed/sync'd after this transaction completes BUT on return this edit must have its region edit/sequence id assigned else it messes up our unification of mvcc and sequenceid. On return key will have the region edit/sequence id filled in.

Specified by:

appendMarker in interface WAL

Parameters:

info - the regioninfo associated with append

key - Modified by this call; we add to it this edits region edit/sequence id.

edits - Edits to append. MAY CONTAIN NO EDITS for case where we want to get an edit sequence id that is after all currently appended edits.

Returns:

Returns a 'transaction id' and key will have the region edit/sequence id in it.

Throws:

IOException

See Also:

• WAL.appendData(RegionInfo, WALKeyImpl, WALEdit)

markFutureDoneAndOffer

protected void markFutureDoneAndOffer(SyncFuture future,
 long txid,
 Throwable t)

Helper that marks the future as DONE and offers it back to the cache.

waitingRoll

private static boolean waitingRoll(int epochAndState)

writerBroken

private static boolean writerBroken(int epochAndState)

epoch

private static int epoch(int epochAndState)

trySetReadyForRolling

private boolean trySetReadyForRolling()

syncFailed

private void syncFailed(long epochWhenSync,
 Throwable error)

onException

private void onException(long epochWhenSync,
 Throwable error)

syncCompleted

private void syncCompleted(long epochWhenSync,
 W writer,
 long processedTxid,
 long startTimeNs)

isHsync

private boolean isHsync(long beginTxid,
 long endTxid)

sync

private void sync(W writer)

getSyncedTxid

protected long getSyncedTxid(long processedTxid,
 long completableFutureResult)

This method is to adapt FSHLog and AsyncFSWAL. For AsyncFSWAL, we use highestProcessedAppendTxid at the point we calling AsyncFSWAL.doWriterSync(org.apache.hadoop.hbase.wal.WALProvider.AsyncWriter, boolean, long) method as successful syncedTxid. For FSHLog, because we use multi-thread SyncRunners, we used the result of CompletableFuture as successful syncedTxid.

doWriterSync

protected abstract CompletableFuture<Long> doWriterSync(W writer,
 boolean shouldUseHsync,
 long txidWhenSyn)

finishSyncLowerThanTxid

private int finishSyncLowerThanTxid(long txid)

finishSync

private int finishSync()

getLastTxid

private static long getLastTxid(Deque<FSWALEntry> queue)

appendAndSync

private void appendAndSync()
                    throws IOException

Throws:

IOException

consume

private void consume()

shouldScheduleConsumer

private boolean shouldScheduleConsumer()

append

protected long append(RegionInfo hri,
 WALKeyImpl key,
 WALEdit edits,
 boolean inMemstore)
               throws IOException

Append a set of edits to the WAL.

The WAL is not flushed/sync'd after this transaction completes BUT on return this edit must have its region edit/sequence id assigned else it messes up our unification of mvcc and sequenceid. On return key will have the region edit/sequence id filled in.

NOTE: This appends, at a time that is usually after this call returns, starts a mvcc transaction by calling 'begin' wherein which we assign this update a sequenceid. At assignment time, we stamp all the passed in Cells inside WALEdit with their sequenceId. You must 'complete' the transaction this mvcc transaction by calling MultiVersionConcurrencyControl#complete(...) or a variant otherwise mvcc will get stuck. Do it in the finally of a try/finally block within which this appends lives and any subsequent operations like sync or update of memstore, etc. Get the WriteEntry to pass mvcc out of the passed in WALKey walKey parameter. Be warned that the WriteEntry is not immediately available on return from this method. It WILL be available subsequent to a sync of this append; otherwise, you will just have to wait on the WriteEntry to get filled in.

Parameters:

hri - the regioninfo associated with append

key - Modified by this call; we add to it this edits region edit/sequence id.

edits - Edits to append. MAY CONTAIN NO EDITS for case where we want to get an edit sequence id that is after all currently appended edits.

inMemstore - Always true except for case where we are writing a region event meta marker edit, for example, a compaction completion record into the WAL or noting a Region Open event. In these cases the entry is just so we can finish an unfinished compaction after a crash when the new Server reads the WAL on recovery, etc. These transition event 'Markers' do not go via the memstore. When memstore is false, we presume a Marker event edit.

Returns:

Returns a 'transaction id' and key will have the region edit/sequence id in it.

Throws:

IOException

doSync

protected void doSync(boolean forceSync)
               throws IOException

Throws:

IOException

doSync

protected void doSync(long txid,
 boolean forceSync)
               throws IOException

Throws:

IOException

drainNonMarkerEditsAndFailSyncs

private void drainNonMarkerEditsAndFailSyncs()

createWriterInstance

protected abstract W createWriterInstance(org.apache.hadoop.fs.FileSystem fs,
 org.apache.hadoop.fs.Path path)
                                   throws IOException,
CommonFSUtils.StreamLacksCapabilityException

Throws:

IOException

CommonFSUtils.StreamLacksCapabilityException

createCombinedWriter

protected abstract W createCombinedWriter(W localWriter,
 W remoteWriter)

waitForSafePoint

protected final void waitForSafePoint()

recoverLease

private void recoverLease(org.apache.hadoop.fs.FileSystem fs,
 org.apache.hadoop.fs.Path p,
 org.apache.hadoop.conf.Configuration conf)

closeWriter

protected final void closeWriter(W writer,
 org.apache.hadoop.fs.Path path)

doReplaceWriter

protected void doReplaceWriter(org.apache.hadoop.fs.Path oldPath,
 org.apache.hadoop.fs.Path newPath,
 W nextWriter)
                        throws IOException

Notice that you need to clear the rollRequested flag in this method, as the new writer will begin to work before returning from this method. If we clear the flag after returning from this call, we may miss a roll request. The implementation class should choose a proper place to clear the rollRequested flag, so we do not miss a roll request, typically before you start writing to the new writer.

Throws:

IOException

onWriterReplaced

protected abstract void onWriterReplaced(W nextWriter)

doShutdown

protected void doShutdown()
                   throws IOException

Throws:

IOException

doCleanUpResources

protected void doCleanUpResources()

doAppend

protected abstract void doAppend(W writer,
 FSWALEntry entry)
                          throws IOException

Throws:

IOException

getPipeline

abstract org.apache.hadoop.hdfs.protocol.DatanodeInfo[] getPipeline()

This method gets the pipeline for the current WAL.

getLogReplication

abstract int getLogReplication()

This method gets the datanode replication count for the current WAL.

doCheckLogLowReplication

protected abstract boolean doCheckLogLowReplication()

isWriterBroken

protected boolean isWriterBroken()

onAppendEntryFailed

private void onAppendEntryFailed(IOException exception)

checkSlowSyncCount

protected void checkSlowSyncCount()

doCheckSlowSync

protected boolean doCheckSlowSync()

Returns true if we exceeded the slow sync roll threshold over the last check interval

checkLogLowReplication

public void checkLogLowReplication(long checkInterval)

skipRemoteWAL

public void skipRemoteWAL(boolean markerEditOnly)

Description copied from interface: WAL

Tell the WAL that when creating new writer you can skip creating the remote writer.

Used by sync replication for switching states from ACTIVE, where the remote cluster is broken.

Specified by:

skipRemoteWAL in interface WAL

split

private static void split(org.apache.hadoop.conf.Configuration conf,
 org.apache.hadoop.fs.Path p)
                   throws IOException

Throws:

IOException

getWriter

W getWriter()

usage

private static void usage()

main

public static void main(String[] args)
                 throws IOException

Pass one or more log file names, and it will either dump out a text version on stdout or split the specified log files.

Throws:

IOException