AbstractFSWAL (Apache HBase 2.5.0 API)

java.lang.Object
- org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL<W>

All Implemented Interfaces:

Closeable, AutoCloseable, WALFileLengthProvider, WAL

Direct Known Subclasses:

AsyncFSWAL, FSHLog
```
@InterfaceAudience.Private
public abstract class AbstractFSWAL<W extends WALProvider.WriterBase>
extends Object
implements WAL
```
Implementation of WAL to go against FileSystem; i.e. keep WALs in HDFS. Only one WAL is ever being written at a time. When a WAL hits a configured maximum size, it is rolled. This is done internal to the implementation.
As data is flushed from the MemStore to other on-disk structures (files sorted by key, hfiles), a WAL becomes obsolete. We can let go of all the log edits/entries for a given HRegion-sequence id. A bunch of work in the below is done keeping account of these region sequence ids -- what is flushed out to hfiles, and what is yet in WAL and in memory only.
It is only practical to delete entire files. Thus, we delete an entire on-disk file F when all of the edits in F have a log-sequence-id that's older (smaller) than the most-recent flush.
To read an WAL, call WALFactory.createReader(org.apache.hadoop.fs.FileSystem, org.apache.hadoop.fs.Path). *
Failure Semantic
If an exception on append or sync, roll the WAL because the current WAL is now a lame duck; any more appends or syncs will fail also with the same original exception. If we have made successful appends to the WAL and we then are unable to sync them, our current semantic is to return error to the client that the appends failed but also to abort the current context, usually the hosting server. We need to replay the WALs.
TODO: Change this semantic. A roll of WAL may be sufficient as long as we have flagged client that the append failed.
TODO: replication may pick up these last edits though they have been marked as failed append (Need to keep our own file lengths, not rely on HDFS).

Nested Class Summary

Nested Classes
Modifier and Type Class and Description

private static class AbstractFSWAL.WalProps
- Nested classes/interfaces inherited from interface org.apache.hadoop.hbase.wal.WAL
  WAL.Entry, WAL.Reader

Nested Classes
Modifier and Type	Class and Description
`private static class`	`AbstractFSWAL.WalProps`

Field Summary

Fields
Modifier and Type	Field and Description
`protected Abortable`	`abortable`
`private int`	`archiveRetries`
`protected long`	`blocksize` Block size to use writing files.
`protected boolean`	`closed`
`protected org.apache.hadoop.conf.Configuration`	`conf` conf object
`protected WALCoprocessorHost`	`coprocessorHost`
`protected static int`	`DEFAULT_ROLL_ON_SYNC_TIME_MS`
`protected static int`	`DEFAULT_SLOW_SYNC_ROLL_INTERVAL_MS`
`protected static int`	`DEFAULT_SLOW_SYNC_ROLL_THRESHOLD`
`protected static int`	`DEFAULT_SLOW_SYNC_TIME_MS`
`protected static int`	`DEFAULT_WAL_SYNC_TIMEOUT_MS`
`protected AtomicLong`	`filenum`
`protected org.apache.hadoop.fs.FileSystem`	`fs` file system instance
`protected AtomicLong`	`highestSyncedTxid` Updated to the transaction id of the last successful sync call.
`protected long`	`highestUnsyncedTxid` The highest known outstanding unsync'd WALEdit transaction id.
`protected String`	`implClassName` The class name of the runtime implementation, used as prefix for logging/tracing.
`protected Map<String,W>`	`inflightWALClosures` Tracks the logs in the process of being closed.
`private long`	`lastTimeCheckLowReplication`
`private long`	`lastTimeCheckSlowSync`
`protected List<WALActionsListener>`	`listeners` Listeners that are called on WAL events.
`private static org.slf4j.Logger`	`LOG`
`(package private) Comparator<org.apache.hadoop.fs.Path>`	`LOG_NAME_COMPARATOR` WAL Comparator; it compares the timestamp (log filenum), present in the log file name.
`private ExecutorService`	`logArchiveExecutor`
`protected long`	`logrollsize`
`static String`	`MAX_LOGS`
`protected int`	`maxLogs`
`protected AtomicInteger`	`numEntries`
`protected org.apache.hadoop.fs.PathFilter`	`ourFiles` Matches just those wal files that belong to this wal instance.
`protected String`	`prefixPathStr` Prefix used when checking for wal membership.
`static String`	`RING_BUFFER_SLOT_COUNT`
`protected static String`	`ROLL_ON_SYNC_TIME_MS`
`protected long`	`rollOnSyncNs`
`protected AtomicBoolean`	`rollRequested`
`protected ReentrantLock`	`rollWriterLock` This lock makes sure only one log roll runs at a time.
`protected SequenceIdAccounting`	`sequenceIdAccounting` Class that does accounting of sequenceids in WAL subsystem.
`protected AtomicBoolean`	`shutdown`
`protected static String`	`SLOW_SYNC_ROLL_INTERVAL_MS`
`protected static String`	`SLOW_SYNC_ROLL_THRESHOLD`
`protected static String`	`SLOW_SYNC_TIME_MS`
`protected int`	`slowSyncCheckInterval`
`protected AtomicInteger`	`slowSyncCount`
`protected long`	`slowSyncNs`
`protected int`	`slowSyncRollThreshold`
`protected SyncFutureCache`	`syncFutureCache` A cache of sync futures reused by threads.
`protected AtomicLong`	`totalLogSize` The total size of wal
`protected boolean`	`useHsync`
`static String`	`WAL_ROLL_MULTIPLIER`
`protected static String`	`WAL_SYNC_TIMEOUT_MS`
`protected org.apache.hadoop.fs.Path`	`walArchiveDir` dir path where old logs are kept.
`protected org.apache.hadoop.fs.Path`	`walDir` WAL directory, where all WAL files would be placed.
`protected ConcurrentNavigableMap<org.apache.hadoop.fs.Path,AbstractFSWAL.WalProps>`	`walFile2Props` Map of WAL log file to properties.
`protected String`	`walFilePrefix` Prefix of a WAL file, usually the region server name it is hosted on.
`protected String`	`walFileSuffix` Suffix included on generated wal file names
`private long`	`walSyncTimeoutNs`
`(package private) W`	`writer` Current log file.

Constructor Summary

Constructors
Modifier	Constructor and Description
`protected`	`AbstractFSWAL(org.apache.hadoop.fs.FileSystem fs, Abortable abortable, org.apache.hadoop.fs.Path rootDir, String logDir, String archiveDir, org.apache.hadoop.conf.Configuration conf, List<WALActionsListener> listeners, boolean failIfWALExists, String prefix, String suffix)`
`protected`	`AbstractFSWAL(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path rootDir, String logDir, String archiveDir, org.apache.hadoop.conf.Configuration conf, List<WALActionsListener> listeners, boolean failIfWALExists, String prefix, String suffix)`

Method Summary

All Methods Static Methods Instance Methods Abstract Methods Concrete Methods
Modifier and Type	Method and Description
`void`	`abortCacheFlush(byte[] encodedRegionName)` Abort a cache flush.
`protected abstract long`	`append(RegionInfo info, WALKeyImpl key, WALEdit edits, boolean inMemstore)` Append a set of edits to the WAL.
`long`	`appendData(RegionInfo info, WALKeyImpl key, WALEdit edits)` Append a set of data edits to the WAL.
`protected boolean`	`appendEntry(W writer, FSWALEntry entry)`
`long`	`appendMarker(RegionInfo info, WALKeyImpl key, WALEdit edits)` Append an operational 'meta' event marker edit to the WAL.
`protected void`	`archive(Pair<org.apache.hadoop.fs.Path,Long> log)`
`protected void`	`archiveLogFile(org.apache.hadoop.fs.Path p)`
`protected void`	`atHeadOfRingBufferEventHandlerAppend()` Exposed for testing only.
`protected void`	`blockOnSync(SyncFuture syncFuture)`
`private int`	`calculateMaxLogFiles(org.apache.hadoop.conf.Configuration conf, long logRollSize)`
`void`	`checkLogLowReplication(long checkInterval)`
`private void`	`cleanOldLogs()` Archive old logs.
`void`	`close()` Caller no longer needs any edits from this WAL.
`void`	`completeCacheFlush(byte[] encodedRegionName, long maxFlushedSeqId)` Complete the cache flush.
`protected org.apache.hadoop.fs.Path`	`computeFilename(long filenum)` This is a convenience method that computes a new filename with a given file-number.
`private IOException`	`convertInterruptedExceptionToIOException(InterruptedException ie)`
`private io.opentelemetry.api.trace.Span`	`createSpan(String name)`
`protected abstract W`	`createWriterInstance(org.apache.hadoop.fs.Path path)`
`protected abstract void`	`doAppend(W writer, FSWALEntry entry)`
`protected abstract boolean`	`doCheckLogLowReplication()`
`protected boolean`	`doCheckSlowSync()` Returns true if we exceeded the slow sync roll threshold over the last check interval
`protected abstract void`	`doReplaceWriter(org.apache.hadoop.fs.Path oldPath, org.apache.hadoop.fs.Path newPath, W nextWriter)` Notice that you need to clear the `rollRequested` flag in this method, as the new writer will begin to work before returning from this method.
`protected abstract void`	`doShutdown()`
`protected abstract void`	`doSync(boolean forceSync)`
`protected abstract void`	`doSync(long txid, boolean forceSync)`
`private static IOException`	`ensureIOException(Throwable t)`
`(package private) Map<byte[],List<byte[]>>`	`findRegionsToForceFlush()` If the number of un-archived WAL files ('live' WALs) is greater than maximum allowed, check the first (oldest) WAL, and return those regions which should be flushed so that it can be let-go/'archived'.
`WALCoprocessorHost`	`getCoprocessorHost()` Returns Coprocessor host.
`org.apache.hadoop.fs.Path`	`getCurrentFileName()` This is a convenience method that computes a new filename with a given using the current WAL file-number n
`long`	`getEarliestMemStoreSeqNum(byte[] encodedRegionName)` Gets the earliest unflushed sequence id in the memstore for the region.
`long`	`getEarliestMemStoreSeqNum(byte[] encodedRegionName, byte[] familyName)` Gets the earliest unflushed sequence id in the memstore for the store.
`long`	`getFilenum()`
`protected long`	`getFileNumFromFileName(org.apache.hadoop.fs.Path fileName)` A log file has a creation timestamp (in ms) in its file name (`filenum`.
`(package private) org.apache.hadoop.fs.FileStatus[]`	`getFiles()` Get the backing files associated with this WAL.
`int`	`getInflightWALCloseCount()` Returns number of WALs currently in the process of closing.
`long`	`getLogFileSize()` Returns the size of log files in use
`OptionalLong`	`getLogFileSizeIfBeingWritten(org.apache.hadoop.fs.Path path)` if the given `path` is being written currently, then return its length.
`(package private) abstract int`	`getLogReplication()` This method gets the datanode replication count for the current WAL.
`private org.apache.hadoop.fs.Path`	`getNewPath()` retrieve the next path to use for writing.
`int`	`getNumLogFiles()` Returns the number of log files in use
`int`	`getNumRolledLogFiles()` Returns the number of rolled log files
`(package private) org.apache.hadoop.fs.Path`	`getOldPath()`
`(package private) abstract org.apache.hadoop.hdfs.protocol.DatanodeInfo[]`	`getPipeline()` This method gets the pipeline for the current WAL.
`protected int`	`getPreallocatedEventCount()`
`protected SyncFuture`	`getSyncFuture(long sequence, boolean forceSync)`
`(package private) long`	`getUnflushedEntriesCount()`
`static org.apache.hadoop.fs.Path`	`getWALArchivePath(org.apache.hadoop.fs.Path archiveDir, org.apache.hadoop.fs.Path p)`
`void`	`init()` Used to initialize the WAL.
`protected boolean`	`isLogRollRequested()`
`(package private) boolean`	`isUnflushedEntries()`
`protected void`	`logRollAndSetupWalProps(org.apache.hadoop.fs.Path oldPath, org.apache.hadoop.fs.Path newPath, long oldFileLen)`
`static void`	`main(String[] args)` Pass one or more log file names and it will either dump out a text version on `stdout` or split the specified log files.
`private long`	`postAppend(WAL.Entry e, long elapsedTime)`
`protected void`	`postSync(long timeInNanos, int handlerSyncs)`
`void`	`registerWALActionsListener(WALActionsListener listener)` Registers WALActionsListener
`(package private) org.apache.hadoop.fs.Path`	`replaceWriter(org.apache.hadoop.fs.Path oldPath, org.apache.hadoop.fs.Path newPath, W nextWriter)` Cleans up current writer closing it and then puts in place the passed in `nextWriter`.
`void`	`requestLogRoll()`
`protected void`	`requestLogRoll(WALActionsListener.RollRequestReason reason)`
`Map<byte[],List<byte[]>>`	`rollWriter()` Roll the log writer.
`Map<byte[],List<byte[]>>`	`rollWriter(boolean force)` Roll the log writer.
`private Map<byte[],List<byte[]>>`	`rollWriterInternal(boolean force)`
`void`	`shutdown()` Stop accepting new writes.
`private static void`	`split(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path p)`
`protected long`	`stampSequenceIdAndPublishToRingBuffer(RegionInfo hri, WALKeyImpl key, WALEdit edits, boolean inMemstore, com.lmax.disruptor.RingBuffer<RingBufferTruck> ringBuffer)`
`Long`	`startCacheFlush(byte[] encodedRegionName, Map<byte[],Long> familyToSeq)`
`Long`	`startCacheFlush(byte[] encodedRegionName, Set<byte[]> families)` WAL keeps track of the sequence numbers that are as yet not flushed im memstores in order to be able to do accounting to figure which WALs can be let go.
`void`	`sync()` Sync what we have in the WAL.
`void`	`sync(boolean forceSync)`
`void`	`sync(long txid)` Sync the WAL if the txId was not already sync'd.
`void`	`sync(long txid, boolean forceSync)`
`private void`	`tellListenersAboutPostLogRoll(org.apache.hadoop.fs.Path oldPath, org.apache.hadoop.fs.Path newPath)` Tell listeners about post log roll.
`private void`	`tellListenersAboutPreLogRoll(org.apache.hadoop.fs.Path oldPath, org.apache.hadoop.fs.Path newPath)` Tell listeners about pre log roll.
`String`	`toString()` Human readable identifying information about the state of this WAL.
`boolean`	`unregisterWALActionsListener(WALActionsListener listener)` Unregisters WALActionsListener
`void`	`updateStore(byte[] encodedRegionName, byte[] familyName, Long sequenceid, boolean onlyIfGreater)` updates the sequence number of a specific store.
`private static void`	`usage()`

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.hadoop.hbase.wal.WAL
getTimestamp

- Field Detail
  - LOG
```
private static final org.slf4j.Logger LOG
```
  - SLOW_SYNC_TIME_MS
```
protected static final String SLOW_SYNC_TIME_MS
```
    See Also:
    
    Constant Field Values
  - DEFAULT_SLOW_SYNC_TIME_MS
```
protected static final int DEFAULT_SLOW_SYNC_TIME_MS
```
    See Also:
    
    Constant Field Values
  - ROLL_ON_SYNC_TIME_MS
```
protected static final String ROLL_ON_SYNC_TIME_MS
```
    See Also:
    
    Constant Field Values
  - DEFAULT_ROLL_ON_SYNC_TIME_MS
```
protected static final int DEFAULT_ROLL_ON_SYNC_TIME_MS
```
    See Also:
    
    Constant Field Values
  - SLOW_SYNC_ROLL_THRESHOLD
```
protected static final String SLOW_SYNC_ROLL_THRESHOLD
```
    See Also:
    
    Constant Field Values
  - DEFAULT_SLOW_SYNC_ROLL_THRESHOLD
```
protected static final int DEFAULT_SLOW_SYNC_ROLL_THRESHOLD
```
    See Also:
    
    Constant Field Values
  - SLOW_SYNC_ROLL_INTERVAL_MS
```
protected static final String SLOW_SYNC_ROLL_INTERVAL_MS
```
    See Also:
    
    Constant Field Values
  - DEFAULT_SLOW_SYNC_ROLL_INTERVAL_MS
```
protected static final int DEFAULT_SLOW_SYNC_ROLL_INTERVAL_MS
```
    See Also:
    
    Constant Field Values
  - WAL_SYNC_TIMEOUT_MS
```
protected static final String WAL_SYNC_TIMEOUT_MS
```
    See Also:
    
    Constant Field Values
  - DEFAULT_WAL_SYNC_TIMEOUT_MS
```
protected static final int DEFAULT_WAL_SYNC_TIMEOUT_MS
```
    See Also:
    
    Constant Field Values
  - WAL_ROLL_MULTIPLIER
```
public static final String WAL_ROLL_MULTIPLIER
```
    See Also:
    
    Constant Field Values
  - MAX_LOGS
```
public static final String MAX_LOGS
```
    See Also:
    
    Constant Field Values
  - RING_BUFFER_SLOT_COUNT
```
public static final String RING_BUFFER_SLOT_COUNT
```
    See Also:
    
    Constant Field Values
  - fs
```
protected final org.apache.hadoop.fs.FileSystem fs
```
    file system instance
  - walDir
```
protected final org.apache.hadoop.fs.Path walDir
```
    WAL directory, where all WAL files would be placed.
  - walArchiveDir
```
protected final org.apache.hadoop.fs.Path walArchiveDir
```
    dir path where old logs are kept.
  - ourFiles
```
protected final org.apache.hadoop.fs.PathFilter ourFiles
```
    Matches just those wal files that belong to this wal instance.
  - walFilePrefix
```
protected final String walFilePrefix
```
    Prefix of a WAL file, usually the region server name it is hosted on.
  - walFileSuffix
```
protected final String walFileSuffix
```
    Suffix included on generated wal file names
  - prefixPathStr
```
protected final String prefixPathStr
```
    Prefix used when checking for wal membership.
  - coprocessorHost
```
protected final WALCoprocessorHost coprocessorHost
```
  - conf
```
protected final org.apache.hadoop.conf.Configuration conf
```
    conf object
  - abortable
```
protected final Abortable abortable
```
  - listeners
```
protected final List<WALActionsListener> listeners
```
    Listeners that are called on WAL events.
  - inflightWALClosures
```
protected final Map<String,W extends WALProvider.WriterBase> inflightWALClosures
```
    Tracks the logs in the process of being closed.
  - sequenceIdAccounting
```
protected final SequenceIdAccounting sequenceIdAccounting
```
    Class that does accounting of sequenceids in WAL subsystem. Holds oldest outstanding sequence id as yet not flushed as well as the most recent edit sequence id appended to the WAL. Has facility for answering questions such as "Is it safe to GC a WAL?".
  - slowSyncNs
```
protected final long slowSyncNs
```
  - rollOnSyncNs
```
protected final long rollOnSyncNs
```
  - slowSyncRollThreshold
```
protected final int slowSyncRollThreshold
```
  - slowSyncCheckInterval
```
protected final int slowSyncCheckInterval
```
  - slowSyncCount
```
protected final AtomicInteger slowSyncCount
```
  - walSyncTimeoutNs
```
private final long walSyncTimeoutNs
```
  - logrollsize
```
protected final long logrollsize
```
  - blocksize
```
protected final long blocksize
```
    Block size to use writing files.
  - maxLogs
```
protected final int maxLogs
```
  - useHsync
```
protected final boolean useHsync
```
  - rollWriterLock
```
protected final ReentrantLock rollWriterLock
```
    This lock makes sure only one log roll runs at a time. Should not be taken while any other lock is held. We don't just use synchronized because that results in bogus and tedious findbugs warning when it thinks synchronized controls writer thread safety. It is held when we are actually rolling the log. It is checked when we are looking to see if we should roll the log or not.
  - filenum
```
protected final AtomicLong filenum
```
  - numEntries
```
protected final AtomicInteger numEntries
```
  - highestUnsyncedTxid
```
protected volatile long highestUnsyncedTxid
```
    The highest known outstanding unsync'd WALEdit transaction id. Usually, we use a queue to pass WALEdit to background consumer thread, and the transaction id is the sequence number of the corresponding entry in queue.
  - highestSyncedTxid
```
protected final AtomicLong highestSyncedTxid
```
    Updated to the transaction id of the last successful sync call. This can be less than highestUnsyncedTxid for case where we have an append where a sync has not yet come in for it.
  - totalLogSize
```
protected final AtomicLong totalLogSize
```
    The total size of wal
  - writer
```
volatile W extends WALProvider.WriterBase writer
```
    Current log file.
  - lastTimeCheckLowReplication
```
private volatile long lastTimeCheckLowReplication
```
  - lastTimeCheckSlowSync
```
private volatile long lastTimeCheckSlowSync
```
  - closed
```
protected volatile boolean closed
```
  - shutdown
```
protected final AtomicBoolean shutdown
```
  - LOG_NAME_COMPARATOR
```
final Comparator<org.apache.hadoop.fs.Path> LOG_NAME_COMPARATOR
```
    WAL Comparator; it compares the timestamp (log filenum), present in the log file name. Throws an IllegalArgumentException if used to compare paths from different wals.
  - walFile2Props
```
protected ConcurrentNavigableMap<org.apache.hadoop.fs.Path,AbstractFSWAL.WalProps> walFile2Props
```
    Map of WAL log file to properties. The map is sorted by the log file creation timestamp (contained in the log file name).
  - syncFutureCache
```
protected final SyncFutureCache syncFutureCache
```
    A cache of sync futures reused by threads.
  - implClassName
```
protected final String implClassName
```
    The class name of the runtime implementation, used as prefix for logging/tracing.
    Performance testing shows getClass().getSimpleName() might be a bottleneck so we store it here, refer to HBASE-17676 for more details
  - rollRequested
```
protected final AtomicBoolean rollRequested
```
  - logArchiveExecutor
```
private final ExecutorService logArchiveExecutor
```
  - archiveRetries
```
private final int archiveRetries
```
- Constructor Detail
  - AbstractFSWAL
```
protected AbstractFSWAL(org.apache.hadoop.fs.FileSystem fs,
                        org.apache.hadoop.fs.Path rootDir,
                        String logDir,
                        String archiveDir,
                        org.apache.hadoop.conf.Configuration conf,
                        List<WALActionsListener> listeners,
                        boolean failIfWALExists,
                        String prefix,
                        String suffix)
                 throws FailedLogCloseException,
                        IOException
```
    Throws:
    
    FailedLogCloseException
    
    IOException
  - AbstractFSWAL
```
protected AbstractFSWAL(org.apache.hadoop.fs.FileSystem fs,
                        Abortable abortable,
                        org.apache.hadoop.fs.Path rootDir,
                        String logDir,
                        String archiveDir,
                        org.apache.hadoop.conf.Configuration conf,
                        List<WALActionsListener> listeners,
                        boolean failIfWALExists,
                        String prefix,
                        String suffix)
                 throws FailedLogCloseException,
                        IOException
```
    Throws:
    
    FailedLogCloseException
    
    IOException
- Method Detail
  - getFilenum
```
public long getFilenum()
```
  - getFileNumFromFileName
```
protected long getFileNumFromFileName(org.apache.hadoop.fs.Path fileName)
```
    A log file has a creation timestamp (in ms) in its file name (filenum. This helper method returns the creation timestamp from a given log file. It extracts the timestamp assuming the filename is created with the computeFilename(long filenum) method.
    
    Returns:
    
    timestamp, as in the log file name.
  - calculateMaxLogFiles
```
private int calculateMaxLogFiles(org.apache.hadoop.conf.Configuration conf,
                                 long logRollSize)
```
  - getPreallocatedEventCount
```
protected final int getPreallocatedEventCount()
```
  - init
```
public void init()
          throws IOException
```
    Used to initialize the WAL. Usually just call rollWriter to create the first log writer.
    
    Throws:
    
    IOException
  - registerWALActionsListener
```
public void registerWALActionsListener(WALActionsListener listener)
```
    Description copied from interface: WAL
    
    Registers WALActionsListener
    
    Specified by:
    
    registerWALActionsListener in interface WAL
  - unregisterWALActionsListener
```
public boolean unregisterWALActionsListener(WALActionsListener listener)
```
    Description copied from interface: WAL
    
    Unregisters WALActionsListener
    
    Specified by:
    
    unregisterWALActionsListener in interface WAL
  - getCoprocessorHost
```
public WALCoprocessorHost getCoprocessorHost()
```
    Description copied from interface: WAL
    
    Returns Coprocessor host.
    
    Specified by:
    
    getCoprocessorHost in interface WAL
  - startCacheFlush
```
public Long startCacheFlush(byte[] encodedRegionName,
                            Set<byte[]> families)
```
    Description copied from interface: WAL
    
    WAL keeps track of the sequence numbers that are as yet not flushed im memstores in order to be able to do accounting to figure which WALs can be let go. This method tells WAL that some region is about to flush. The flush can be the whole region or for a column family of the region only.
    Currently, it is expected that the update lock is held for the region; i.e. no concurrent appends while we set up cache flush.
    
    Specified by:
    
    startCacheFlush in interface WAL
    
    families - Families to flush. May be a subset of all families in the region.
    
    Returns:
    
    Returns HConstants.NO_SEQNUM if we are flushing the whole region OR if we are flushing a subset of all families but there are no edits in those families not being flushed; in other words, this is effectively same as a flush of all of the region though we were passed a subset of regions. Otherwise, it returns the sequence id of the oldest/lowest outstanding edit.
    
    See Also:
    
    WAL.completeCacheFlush(byte[], long), WAL.abortCacheFlush(byte[])
  - startCacheFlush
```
public Long startCacheFlush(byte[] encodedRegionName,
                            Map<byte[],Long> familyToSeq)
```
    Specified by:
    
    startCacheFlush in interface WAL
  - completeCacheFlush
```
public void completeCacheFlush(byte[] encodedRegionName,
                               long maxFlushedSeqId)
```
    Description copied from interface: WAL
    
    Complete the cache flush.
    
    Specified by:
    
    completeCacheFlush in interface WAL
    
    Parameters:
    
    encodedRegionName - Encoded region name.
    
    maxFlushedSeqId - The maxFlushedSeqId for this flush. There is no edit in memory that is less that this sequence id.
    
    See Also:
    
    WAL.startCacheFlush(byte[], Set), WAL.abortCacheFlush(byte[])
  - abortCacheFlush
```
public void abortCacheFlush(byte[] encodedRegionName)
```
    Description copied from interface: WAL
    
    Abort a cache flush. Call if the flush fails. Note that the only recovery for an aborted flush currently is a restart of the regionserver so the snapshot content dropped by the failure gets restored to the memstore.
    
    Specified by:
    
    abortCacheFlush in interface WAL
    
    Parameters:
    
    encodedRegionName - Encoded region name.
  - getEarliestMemStoreSeqNum
```
public long getEarliestMemStoreSeqNum(byte[] encodedRegionName)
```
    Description copied from interface: WAL
    
    Gets the earliest unflushed sequence id in the memstore for the region.
    
    Specified by:
    
    getEarliestMemStoreSeqNum in interface WAL
    
    Parameters:
    
    encodedRegionName - The region to get the number for.
    
    Returns:
    
    The earliest/lowest/oldest sequence id if present, HConstants.NO_SEQNUM if absent.
  - getEarliestMemStoreSeqNum
```
public long getEarliestMemStoreSeqNum(byte[] encodedRegionName,
                                      byte[] familyName)
```
    Description copied from interface: WAL
    
    Gets the earliest unflushed sequence id in the memstore for the store.
    
    Specified by:
    
    getEarliestMemStoreSeqNum in interface WAL
    
    Parameters:
    
    encodedRegionName - The region to get the number for.
    
    familyName - The family to get the number for.
    
    Returns:
    
    The earliest/lowest/oldest sequence id if present, HConstants.NO_SEQNUM if absent.
  - rollWriter
```
public Map<byte[],List<byte[]>> rollWriter()
                                    throws FailedLogCloseException,
                                           IOException
```
    Description copied from interface: WAL
    
    Roll the log writer. That is, start writing log messages to a new file.
    The implementation is synchronized in order to make sure there's one rollWriter running at any given time.
    
    Specified by:
    
    rollWriter in interface WAL
    
    Returns:
    
    If lots of logs, flush the stores of returned regions so next time through we can clean logs. Returns null if nothing to flush. Names are actual region names as returned by RegionInfo.getEncodedName()
    
    Throws:
    
    FailedLogCloseException
    
    IOException
  - sync
```
public final void sync()
                throws IOException
```
    Description copied from interface: WAL
    
    Sync what we have in the WAL.
    
    Specified by:
    
    sync in interface WAL
    
    Throws:
    
    IOException
  - sync
```
public final void sync(long txid)
                throws IOException
```
    Description copied from interface: WAL
    
    Sync the WAL if the txId was not already sync'd.
    
    Specified by:
    
    sync in interface WAL
    
    Parameters:
    
    txid - Transaction id to sync to.
    
    Throws:
    
    IOException
  - sync
```
public final void sync(boolean forceSync)
                throws IOException
```
    Specified by:
    
    sync in interface WAL
    
    Parameters:
    
    forceSync - Flag to force sync rather than flushing to the buffer. Example - Hadoop hflush vs hsync.
    
    Throws:
    
    IOException
  - sync
```
public final void sync(long txid,
                       boolean forceSync)
                throws IOException
```
    Specified by:
    
    sync in interface WAL
    
    Parameters:
    
    txid - Transaction id to sync to.
    
    forceSync - Flag to force sync rather than flushing to the buffer. Example - Hadoop hflush vs hsync.
    
    Throws:
    
    IOException
  - doSync
```
protected abstract void doSync(boolean forceSync)
                        throws IOException
```
    Throws:
    
    IOException
  - doSync
```
protected abstract void doSync(long txid,
                               boolean forceSync)
                        throws IOException
```
    Throws:
    
    IOException
  - computeFilename
```
protected org.apache.hadoop.fs.Path computeFilename(long filenum)
```
    This is a convenience method that computes a new filename with a given file-number.
    
    Parameters:
    
    filenum - to use n
  - getCurrentFileName
```
public org.apache.hadoop.fs.Path getCurrentFileName()
```
    This is a convenience method that computes a new filename with a given using the current WAL file-number n
  - getNewPath
```
private org.apache.hadoop.fs.Path getNewPath()
                                      throws IOException
```
    retrieve the next path to use for writing. Increments the internal filenum.
    
    Throws:
    
    IOException
  - getOldPath
```
org.apache.hadoop.fs.Path getOldPath()
```
  - tellListenersAboutPreLogRoll
```
private void tellListenersAboutPreLogRoll(org.apache.hadoop.fs.Path oldPath,
                                          org.apache.hadoop.fs.Path newPath)
                                   throws IOException
```
    Tell listeners about pre log roll.
    
    Throws:
    
    IOException
  - tellListenersAboutPostLogRoll
```
private void tellListenersAboutPostLogRoll(org.apache.hadoop.fs.Path oldPath,
                                           org.apache.hadoop.fs.Path newPath)
                                    throws IOException
```
    Tell listeners about post log roll.
    
    Throws:
    
    IOException
  - getNumRolledLogFiles
```
public int getNumRolledLogFiles()
```
    Returns the number of rolled log files
  - getNumLogFiles
```
public int getNumLogFiles()
```
    Returns the number of log files in use
  - findRegionsToForceFlush
```
Map<byte[],List<byte[]>> findRegionsToForceFlush()
                                          throws IOException
```
    If the number of un-archived WAL files ('live' WALs) is greater than maximum allowed, check the first (oldest) WAL, and return those regions which should be flushed so that it can be let-go/'archived'.
    
    Returns:
    
    stores of regions (encodedRegionNames) to flush in order to archive oldest WAL file.
    
    Throws:
    
    IOException
  - cleanOldLogs
```
private void cleanOldLogs()
                   throws IOException
```
    Archive old logs. A WAL is eligible for archiving if all its WALEdits have been flushed.
    
    Throws:
    
    IOException
  - archive
```
protected void archive(Pair<org.apache.hadoop.fs.Path,Long> log)
```
  - getWALArchivePath
```
public static org.apache.hadoop.fs.Path getWALArchivePath(org.apache.hadoop.fs.Path archiveDir,
                                                          org.apache.hadoop.fs.Path p)
```
  - archiveLogFile
```
protected void archiveLogFile(org.apache.hadoop.fs.Path p)
                       throws IOException
```
    Throws:
    
    IOException
  - logRollAndSetupWalProps
```
protected final void logRollAndSetupWalProps(org.apache.hadoop.fs.Path oldPath,
                                             org.apache.hadoop.fs.Path newPath,
                                             long oldFileLen)
```
  - createSpan
```
private io.opentelemetry.api.trace.Span createSpan(String name)
```
  - replaceWriter
```
org.apache.hadoop.fs.Path replaceWriter(org.apache.hadoop.fs.Path oldPath,
                                        org.apache.hadoop.fs.Path newPath,
                                        W nextWriter)
                                 throws IOException
```
    Cleans up current writer closing it and then puts in place the passed in nextWriter.
    - In the case of creating a new WAL, oldPath will be null.
    - In the case of rolling over from one file to the next, none of the parameters will be null.
    - In the case of closing out this FSHLog with no further use newPath and nextWriter will be null.
    Parameters:
    
    oldPath - may be null
    
    newPath - may be null
    
    nextWriter - may be null
    
    Returns:
    
    the passed in newPath
    
    Throws:
    
    IOException - if there is a problem flushing or closing the underlying FS
  - blockOnSync
```
protected final void blockOnSync(SyncFuture syncFuture)
                          throws IOException
```
    Throws:
    
    IOException
  - ensureIOException
```
private static IOException ensureIOException(Throwable t)
```
  - convertInterruptedExceptionToIOException
```
private IOException convertInterruptedExceptionToIOException(InterruptedException ie)
```
  - rollWriterInternal
```
private Map<byte[],List<byte[]>> rollWriterInternal(boolean force)
                                             throws IOException
```
    Throws:
    
    IOException
  - rollWriter
```
public Map<byte[],List<byte[]>> rollWriter(boolean force)
                                    throws IOException
```
    Description copied from interface: WAL
    
    Roll the log writer. That is, start writing log messages to a new file.
    The implementation is synchronized in order to make sure there's one rollWriter running at any given time. n * If true, force creation of a new writer even if no entries have been written to the current writer
    
    Specified by:
    
    rollWriter in interface WAL
    
    Returns:
    
    If lots of logs, flush the stores of returned regions so next time through we can clean logs. Returns null if nothing to flush. Names are actual region names as returned by RegionInfo.getEncodedName()
    
    Throws:
    
    IOException
  - getLogFileSize
```
public long getLogFileSize()
```
    Returns the size of log files in use
  - requestLogRoll
```
public void requestLogRoll()
```
  - getFiles
```
org.apache.hadoop.fs.FileStatus[] getFiles()
                                    throws IOException
```
    Get the backing files associated with this WAL.
    
    Returns:
    
    may be null if there are no files.
    
    Throws:
    
    IOException
  - shutdown
```
public void shutdown()
              throws IOException
```
    Description copied from interface: WAL
    
    Stop accepting new writes. If we have unsynced writes still in buffer, sync them. Extant edits are left in place in backing storage to be replayed later.
    
    Specified by:
    
    shutdown in interface WAL
    
    Throws:
    
    IOException
  - close
```
public void close()
           throws IOException
```
    Description copied from interface: WAL
    
    Caller no longer needs any edits from this WAL. Implementers are free to reclaim underlying resources after this call; i.e. filesystem based WALs can archive or delete files.
    
    Specified by:
    
    close in interface Closeable
    
    Specified by:
    
    close in interface AutoCloseable
    
    Specified by:
    
    close in interface WAL
    
    Throws:
    
    IOException
  - getInflightWALCloseCount
```
public int getInflightWALCloseCount()
```
    Returns number of WALs currently in the process of closing.
  - updateStore
```
public void updateStore(byte[] encodedRegionName,
                        byte[] familyName,
                        Long sequenceid,
                        boolean onlyIfGreater)
```
    updates the sequence number of a specific store. depending on the flag: replaces current seq number if the given seq id is bigger, or even if it is lower than existing one
    
    Specified by:
    
    updateStore in interface WAL
  - getSyncFuture
```
protected final SyncFuture getSyncFuture(long sequence,
                                         boolean forceSync)
```
  - isLogRollRequested
```
protected boolean isLogRollRequested()
```
  - requestLogRoll
```
protected final void requestLogRoll(WALActionsListener.RollRequestReason reason)
```
  - getUnflushedEntriesCount
```
long getUnflushedEntriesCount()
```
  - isUnflushedEntries
```
boolean isUnflushedEntries()
```
  - atHeadOfRingBufferEventHandlerAppend
```
protected void atHeadOfRingBufferEventHandlerAppend()
```
    Exposed for testing only. Use to tricks like halt the ring buffer appending.
  - appendEntry
```
protected final boolean appendEntry(W writer,
                                    FSWALEntry entry)
                             throws IOException
```
    Throws:
    
    IOException
  - postAppend
```
private long postAppend(WAL.Entry e,
                        long elapsedTime)
                 throws IOException
```
    Throws:
    
    IOException
  - postSync
```
protected final void postSync(long timeInNanos,
                              int handlerSyncs)
```
  - stampSequenceIdAndPublishToRingBuffer
```
protected final long stampSequenceIdAndPublishToRingBuffer(RegionInfo hri,
                                                           WALKeyImpl key,
                                                           WALEdit edits,
                                                           boolean inMemstore,
                                                           com.lmax.disruptor.RingBuffer<RingBufferTruck> ringBuffer)
                                                    throws IOException
```
    Throws:
    
    IOException
  - toString
```
public String toString()
```
    Description copied from interface: WAL
    
    Human readable identifying information about the state of this WAL. Implementors are encouraged to include information appropriate for debugging. Consumers are advised not to rely on the details of the returned String; it does not have a defined structure.
    
    Specified by:
    
    toString in interface WAL
    
    Overrides:
    
    toString in class Object
  - getLogFileSizeIfBeingWritten
```
public OptionalLong getLogFileSizeIfBeingWritten(org.apache.hadoop.fs.Path path)
```
    if the given path is being written currently, then return its length.
    This is used by replication to prevent replicating unacked log entries. See https://issues.apache.org/jira/browse/HBASE-14004 for more details.
    
    Specified by:
    
    getLogFileSizeIfBeingWritten in interface WALFileLengthProvider
  - appendData
```
public long appendData(RegionInfo info,
                       WALKeyImpl key,
                       WALEdit edits)
                throws IOException
```
    Description copied from interface: WAL
    
    Append a set of data edits to the WAL. 'Data' here means that the content in the edits will also have transitioned through the memstore.
    The WAL is not flushed/sync'd after this transaction completes BUT on return this edit must have its region edit/sequence id assigned else it messes up our unification of mvcc and sequenceid. On return key will have the region edit/sequence id filled in.
    
    Specified by:
    
    appendData in interface WAL
    
    Parameters:
    
    info - the regioninfo associated with append
    
    key - Modified by this call; we add to it this edits region edit/sequence id.
    
    edits - Edits to append. MAY CONTAIN NO EDITS for case where we want to get an edit sequence id that is after all currently appended edits.
    
    Returns:
    
    Returns a 'transaction id' and key will have the region edit/sequence id in it.
    
    Throws:
    
    IOException
    
    See Also:
    
    WAL.appendMarker(RegionInfo, WALKeyImpl, WALEdit)
  - appendMarker
```
public long appendMarker(RegionInfo info,
                         WALKeyImpl key,
                         WALEdit edits)
                  throws IOException
```
    Description copied from interface: WAL
    
    Append an operational 'meta' event marker edit to the WAL. A marker meta edit could be a FlushDescriptor, a compaction marker, or a region event marker; e.g. region open or region close. The difference between a 'marker' append and a 'data' append as in WAL.appendData(RegionInfo, WALKeyImpl, WALEdit)is that a marker will not have transitioned through the memstore.
    The WAL is not flushed/sync'd after this transaction completes BUT on return this edit must have its region edit/sequence id assigned else it messes up our unification of mvcc and sequenceid. On return key will have the region edit/sequence id filled in.
    
    Specified by:
    
    appendMarker in interface WAL
    
    Parameters:
    
    info - the regioninfo associated with append
    
    key - Modified by this call; we add to it this edits region edit/sequence id.
    
    edits - Edits to append. MAY CONTAIN NO EDITS for case where we want to get an edit sequence id that is after all currently appended edits.
    
    Returns:
    
    Returns a 'transaction id' and key will have the region edit/sequence id in it.
    
    Throws:
    
    IOException
    
    See Also:
    
    WAL.appendData(RegionInfo, WALKeyImpl, WALEdit)
  - append
```
protected abstract long append(RegionInfo info,
                               WALKeyImpl key,
                               WALEdit edits,
                               boolean inMemstore)
                        throws IOException
```
    Append a set of edits to the WAL.
    The WAL is not flushed/sync'd after this transaction completes BUT on return this edit must have its region edit/sequence id assigned else it messes up our unification of mvcc and sequenceid. On return key will have the region edit/sequence id filled in.
    NOTE: This append, at a time that is usually after this call returns, starts an mvcc transaction by calling 'begin' wherein which we assign this update a sequenceid. At assignment time, we stamp all the passed in Cells inside WALEdit with their sequenceId. You must 'complete' the transaction this mvcc transaction by calling MultiVersionConcurrencyControl#complete(...) or a variant otherwise mvcc will get stuck. Do it in the finally of a try/finally block within which this append lives and any subsequent operations like sync or update of memstore, etc. Get the WriteEntry to pass mvcc out of the passed in WALKey walKey parameter. Be warned that the WriteEntry is not immediately available on return from this method. It WILL be available subsequent to a sync of this append; otherwise, you will just have to wait on the WriteEntry to get filled in.
    
    Parameters:
    
    info - the regioninfo associated with append
    
    key - Modified by this call; we add to it this edits region edit/sequence id.
    
    edits - Edits to append. MAY CONTAIN NO EDITS for case where we want to get an edit sequence id that is after all currently appended edits.
    
    inMemstore - Always true except for case where we are writing a region event meta marker edit, for example, a compaction completion record into the WAL or noting a Region Open event. In these cases the entry is just so we can finish an unfinished compaction after a crash when the new Server reads the WAL on recovery, etc. These transition event 'Markers' do not go via the memstore. When memstore is false, we presume a Marker event edit.
    
    Returns:
    
    Returns a 'transaction id' and key will have the region edit/sequence id in it.
    
    Throws:
    
    IOException
  - doAppend
```
protected abstract void doAppend(W writer,
                                 FSWALEntry entry)
                          throws IOException
```
    Throws:
    
    IOException
  - createWriterInstance
```
protected abstract W createWriterInstance(org.apache.hadoop.fs.Path path)
                                   throws IOException,
                                          CommonFSUtils.StreamLacksCapabilityException
```
    Throws:
    
    IOException
    
    CommonFSUtils.StreamLacksCapabilityException
  - doReplaceWriter
```
protected abstract void doReplaceWriter(org.apache.hadoop.fs.Path oldPath,
                                        org.apache.hadoop.fs.Path newPath,
                                        W nextWriter)
                                 throws IOException
```
    Notice that you need to clear the rollRequested flag in this method, as the new writer will begin to work before returning from this method. If we clear the flag after returning from this call, we may miss a roll request. The implementation class should choose a proper place to clear the rollRequested flag so we do not miss a roll request, typically before you start writing to the new writer.
    
    Throws:
    
    IOException
  - doShutdown
```
protected abstract void doShutdown()
                            throws IOException
```
    Throws:
    
    IOException
  - doCheckLogLowReplication
```
protected abstract boolean doCheckLogLowReplication()
```
  - doCheckSlowSync
```
protected boolean doCheckSlowSync()
```
    Returns true if we exceeded the slow sync roll threshold over the last check interval
  - checkLogLowReplication
```
public void checkLogLowReplication(long checkInterval)
```
  - getPipeline
```
abstract org.apache.hadoop.hdfs.protocol.DatanodeInfo[] getPipeline()
```
    This method gets the pipeline for the current WAL.
  - getLogReplication
```
abstract int getLogReplication()
```
    This method gets the datanode replication count for the current WAL.
  - split
```
private static void split(org.apache.hadoop.conf.Configuration conf,
                          org.apache.hadoop.fs.Path p)
                   throws IOException
```
    Throws:
    
    IOException
  - usage
```
private static void usage()
```
  - main
```
public static void main(String[] args)
                 throws IOException
```
    Pass one or more log file names and it will either dump out a text version on stdout or split the specified log files.
    
    Throws:
    
    IOException

Class AbstractFSWAL<W extends WALProvider.WriterBase>

Failure Semantic

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hadoop.hbase.wal.WAL

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Methods inherited from interface org.apache.hadoop.hbase.wal.WAL

Field Detail

LOG

SLOW_SYNC_TIME_MS

DEFAULT_SLOW_SYNC_TIME_MS

ROLL_ON_SYNC_TIME_MS

DEFAULT_ROLL_ON_SYNC_TIME_MS

SLOW_SYNC_ROLL_THRESHOLD

DEFAULT_SLOW_SYNC_ROLL_THRESHOLD

SLOW_SYNC_ROLL_INTERVAL_MS

DEFAULT_SLOW_SYNC_ROLL_INTERVAL_MS

WAL_SYNC_TIMEOUT_MS

DEFAULT_WAL_SYNC_TIMEOUT_MS

WAL_ROLL_MULTIPLIER

MAX_LOGS

RING_BUFFER_SLOT_COUNT

fs

walDir

walArchiveDir

ourFiles

walFilePrefix

walFileSuffix

prefixPathStr

coprocessorHost

conf

abortable

listeners

inflightWALClosures

sequenceIdAccounting

slowSyncNs

rollOnSyncNs

slowSyncRollThreshold

slowSyncCheckInterval

slowSyncCount

walSyncTimeoutNs

logrollsize

blocksize

maxLogs

useHsync

rollWriterLock

filenum

numEntries

highestUnsyncedTxid

highestSyncedTxid

totalLogSize

writer

lastTimeCheckLowReplication

lastTimeCheckSlowSync

closed

shutdown

LOG_NAME_COMPARATOR

walFile2Props

syncFutureCache

implClassName

rollRequested

logArchiveExecutor

archiveRetries

Constructor Detail

AbstractFSWAL

AbstractFSWAL

Method Detail

getFilenum

getFileNumFromFileName

calculateMaxLogFiles

getPreallocatedEventCount

init

registerWALActionsListener

unregisterWALActionsListener

getCoprocessorHost

startCacheFlush

startCacheFlush

completeCacheFlush