FSHLog (Apache HBase 1.2.12 API)

java.lang.Object
- org.apache.hadoop.hbase.regionserver.wal.FSHLog

All Implemented Interfaces:

Closeable, AutoCloseable, WAL
```
@InterfaceAudience.Private
public class FSHLog
extends Object
implements WAL
```
Implementation of WAL to go against FileSystem; i.e. keep WALs in HDFS. Only one WAL is ever being written at a time. When a WAL hits a configured maximum size, it is rolled. This is done internal to the implementation.
As data is flushed from the MemStore to other on-disk structures (files sorted by key, hfiles), a WAL becomes obsolete. We can let go of all the log edits/entries for a given HRegion-sequence id. A bunch of work in the below is done keeping account of these region sequence ids -- what is flushed out to hfiles, and what is yet in WAL and in memory only.
It is only practical to delete entire files. Thus, we delete an entire on-disk file F when all of the edits in F have a log-sequence-id that's older (smaller) than the most-recent flush.
To read an WAL, call WALFactory.createReader(org.apache.hadoop.fs.FileSystem, org.apache.hadoop.fs.Path).
Failure Semantic
If an exception on append or sync, roll the WAL because the current WAL is now a lame duck; any more appends or syncs will fail also with the same original exception. If we have made successful appends to the WAL and we then are unable to sync them, our current semantic is to return error to the client that the appends failed but also to abort the current context, usually the hosting server. We need to replay the WALs. TODO: Change this semantic. A roll of WAL may be sufficient as long as we have flagged client that the append failed. TODO: replication may pick up these last edits though they have been marked as failed append (Need to keep our own file lengths, not rely on HDFS).

Nested Class Summary

Nested Classes
Modifier and Type	Class and Description
`(package private) class`	`FSHLog.RingBufferEventHandler` Handler that is run by the disruptor ringbuffer consumer.
`(package private) static class`	`FSHLog.RingBufferExceptionHandler` Exception handler to pass the disruptor ringbuffer.
`(package private) static class`	`FSHLog.SafePointZigZagLatch` This class is used coordinating two threads holding one thread at a 'safe point' while the orchestrating thread does some work that requires the first thread paused: e.g.
`private class`	`FSHLog.SyncRunner` Thread to runs the hdfs sync call.

Nested classes/interfaces inherited from interface org.apache.hadoop.hbase.wal.WAL
WAL.Entry, WAL.Reader

Field Summary

Fields
Modifier and Type	Field and Description
`private ExecutorService`	`appendExecutor` An executorservice that runs the disruptor AppendEventHandler append executor.
`private NavigableMap<org.apache.hadoop.fs.Path,Map<byte[],Long>>`	`byWalRegionSequenceIds` Map of WAL log file to the latest sequence ids of all regions it has entries of.
`private ThreadLocal<SyncFuture>`	`cachedSyncFutures` Map of `SyncFuture`s owned by Thread objects.
`private DrainBarrier`	`closeBarrier` The barrier used to ensure that close() waits for all log rolls and flushes to finish.
`private boolean`	`closed`
`private AtomicInteger`	`closeErrorCount`
`private int`	`closeErrorsTolerated` Number of log close errors tolerated before we abort
`protected org.apache.hadoop.conf.Configuration`	`conf` conf object
`private AtomicInteger`	`consecutiveLogRolls`
`private WALCoprocessorHost`	`coprocessorHost`
`private static int`	`DEFAULT_SLOW_SYNC_TIME_MS`
`private static int`	`DEFAULT_WAL_SYNC_TIMEOUT_MS`
`private com.lmax.disruptor.dsl.Disruptor<RingBufferTruck>`	`disruptor` The nexus at which all incoming handlers meet.
`private AtomicLong`	`filenum`
`static long`	`FIXED_OVERHEAD`
`protected org.apache.hadoop.fs.FileSystem`	`fs` file system instance
`private org.apache.hadoop.fs.Path`	`fullPathArchiveDir` dir path where old logs are kept.
`private org.apache.hadoop.fs.Path`	`fullPathLogDir` WAL directory, where all WAL files would be placed.
`private org.apache.hadoop.fs.FSDataOutputStream`	`hdfs_out` FSDataOutputStream associated with the current SequenceFile.writer
`private AtomicLong`	`highestSyncedSequence` Updated to the ring buffer sequence of the last successful sync call.
`private long`	`highestUnsyncedSequence` The highest known outstanding unsync'd WALEdit sequence number where sequence number is the ring buffer sequence.
`private long`	`lastTimeCheckLowReplication`
`private List<WALActionsListener>`	`listeners` Listeners that are called on WAL events.
`private static org.apache.commons.logging.Log`	`LOG`
`(package private) Comparator<org.apache.hadoop.fs.Path>`	`LOG_NAME_COMPARATOR` WAL Comparator; it compares the timestamp (log filenum), present in the log file name.
`private String`	`logFilePrefix` Prefix of a WAL file, usually the region server name it is hosted on.
`private String`	`logFileSuffix` Suffix included on generated wal file names
`private long`	`logrollsize`
`private boolean`	`lowReplicationRollEnabled`
`private int`	`lowReplicationRollLimit`
`private int`	`maxLogs`
`private int`	`minTolerableReplication`
`private AtomicInteger`	`numEntries`
`private org.apache.hadoop.fs.PathFilter`	`ourFiles` Matches just those wal files that belong to this wal instance.
`private String`	`prefixPathStr` Prefix used when checking for wal membership.
`private FSHLog.RingBufferEventHandler`	`ringBufferEventHandler` This fellow is run by the above appendExecutor service but it is all about batching up appends and syncs; it may shutdown without cleaning out the last few appends or syncs.
`private ReentrantLock`	`rollWriterLock` This lock makes sure only one log roll runs at a time.
`private SequenceIdAccounting`	`sequenceIdAccounting` Class that does accounting of sequenceids in WAL subsystem.
`private AtomicBoolean`	`shutdown`
`private int`	`slowSyncNs`
`private AtomicLong`	`totalLogSize` The total size of wal
`private long`	`walSyncTimeout`
`(package private) WALProvider.Writer`	`writer` Current log file.

Constructor Summary

Constructors
Constructor and Description
`FSHLog(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path root, String logDir, org.apache.hadoop.conf.Configuration conf)` Constructor.
`FSHLog(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path rootDir, String logDir, String archiveDir, org.apache.hadoop.conf.Configuration conf, List<WALActionsListener> listeners, boolean failIfWALExists, String prefix, String suffix)` Create an edit log at the given `dir` location.

Method Summary

Methods
Modifier and Type	Method and Description
`void`	`abortCacheFlush(byte[] encodedRegionName)` Abort a cache flush.
`protected void`	`afterCreatingZigZagLatch()` Used to manufacture race condition reliably.
`long`	`append(HTableDescriptor htd, HRegionInfo hri, WALKey key, WALEdit edits, boolean inMemstore)` Append a set of edits to the WAL.
`private void`	`archiveLogFile(org.apache.hadoop.fs.Path p)`
`(package private) void`	`atHeadOfRingBufferEventHandlerAppend()` Exposed for testing only.
`protected void`	`beforeWaitOnSafePoint()`
`private org.apache.htrace.Span`	`blockOnSync(SyncFuture syncFuture)`
`private int`	`calculateMaxLogFiles(float memstoreSizeRatio, long logRollSize)`
`void`	`checkLogRoll()` Schedule a log roll if needed.
`private boolean`	`checkLowReplication()`
`private void`	`cleanOldLogs()` Archive old logs.
`void`	`close()` Caller no longer needs any edits from this WAL.
`void`	`completeCacheFlush(byte[] encodedRegionName)` Complete the cache flush.
`protected org.apache.hadoop.fs.Path`	`computeFilename(long filenum)` This is a convenience method that computes a new filename with a given file-number.
`private IOException`	`convertInterruptedExceptionToIOException(InterruptedException ie)`
`protected WALProvider.Writer`	`createWriterInstance(org.apache.hadoop.fs.Path path)` This method allows subclasses to inject different writers without having to extend other methods like rollWriter().
`private static IOException`	`ensureIOException(Throwable t)`
`(package private) byte[][]`	`findRegionsToForceFlush()` If the number of un-archived WAL files is greater than maximum allowed, check the first (oldest) WAL file, and returns those regions which should be flushed so that it can be archived.
`WALCoprocessorHost`	`getCoprocessorHost()`
`org.apache.hadoop.fs.Path`	`getCurrentFileName()` This is a convenience method that computes a new filename with a given using the current WAL file-number
`long`	`getEarliestMemstoreSeqNum(byte[] encodedRegionName)` Gets the earliest unflushed sequence id in the memstore for the region.
`long`	`getEarliestMemstoreSeqNum(byte[] encodedRegionName, byte[] familyName)` Gets the earliest unflushed sequence id in the memstore for the store.
`protected long`	`getFileNumFromFileName(org.apache.hadoop.fs.Path fileName)` A log file has a creation timestamp (in ms) in its file name (`filenum`.
`protected org.apache.hadoop.fs.FileStatus[]`	`getFiles()` Get the backing files associated with this WAL.
`long`	`getLastTimeCheckLowReplication()`
`long`	`getLogFileSize()`
`(package private) int`	`getLogReplication()` This method gets the datanode replication count for the current WAL.
`private org.apache.hadoop.fs.Path`	`getNewPath()` retrieve the next path to use for writing.
`int`	`getNumLogFiles()`
`int`	`getNumRolledLogFiles()`
`(package private) org.apache.hadoop.fs.Path`	`getOldPath()`
`(package private) OutputStream`	`getOutputStream()` Currently, we need to expose the writer's OutputStream to tests so that they can manipulate the default behavior (such as setting the maxRecoveryErrorCount value for example (see `TestWALReplay.testReplayEditsWrittenIntoWAL()`).
`(package private) org.apache.hadoop.hdfs.protocol.DatanodeInfo[]`	`getPipeLine()` This method gets the pipeline for the current WAL.
`private long`	`getSequenceOnRingBuffer()`
`private SyncFuture`	`getSyncFuture(long sequence, org.apache.htrace.Span span)`
`(package private) long`	`getUnflushedEntriesCount()`
`static org.apache.hadoop.fs.Path`	`getWALArchivePath(org.apache.hadoop.fs.Path archiveDir, org.apache.hadoop.fs.Path p)`
`(package private) boolean`	`isLowReplicationRollEnabled()`
`(package private) boolean`	`isUnflushedEntries()`
`static void`	`main(String[] args)` Pass one or more log file names and it will either dump out a text version on `stdout` or split the specified log files.
`private long`	`postAppend(WAL.Entry e, long elapsedTime)`
`private void`	`postSync(long timeInNanos, int handlerSyncs)`
`private void`	`preemptiveSync(ProtobufLogWriter nextWriter)` Run a sync after opening to set up the pipeline.
`private SyncFuture`	`publishSyncOnRingBuffer(long sequence)`
`private SyncFuture`	`publishSyncOnRingBuffer(long sequence, org.apache.htrace.Span span)`
`private SyncFuture`	`publishSyncOnRingBuffer(org.apache.htrace.Span span)`
`private org.apache.htrace.Span`	`publishSyncThenBlockOnCompletion(org.apache.htrace.Span span)`
`void`	`registerWALActionsListener(WALActionsListener listener)` Registers WALActionsListener
`(package private) org.apache.hadoop.fs.Path`	`replaceWriter(org.apache.hadoop.fs.Path oldPath, org.apache.hadoop.fs.Path newPath, WALProvider.Writer nextWriter, org.apache.hadoop.fs.FSDataOutputStream nextHdfsOut)` Cleans up current writer closing it and then puts in place the passed in `nextWriter`.
`void`	`requestLogRoll()`
`private void`	`requestLogRoll(boolean tooFewReplicas)`
`byte[][]`	`rollWriter()` Roll the log writer.
`byte[][]`	`rollWriter(boolean force)` Roll the log writer.
`void`	`shutdown()` Stop accepting new writes.
`private static void`	`split(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path p)`
`Long`	`startCacheFlush(byte[] encodedRegionName, Set<byte[]> families)` WAL keeps track of the sequence numbers that are as yet not flushed im memstores in order to be able to do accounting to figure which WALs can be let go.
`void`	`sync()` Sync what we have in the WAL.
`void`	`sync(long txid)` Sync the WAL if the txId was not already sync'd.
`private void`	`tellListenersAboutPostLogRoll(org.apache.hadoop.fs.Path oldPath, org.apache.hadoop.fs.Path newPath)` Tell listeners about post log roll.
`private void`	`tellListenersAboutPreLogRoll(org.apache.hadoop.fs.Path oldPath, org.apache.hadoop.fs.Path newPath)` Tell listeners about pre log roll.
`String`	`toString()` Human readable identifying information about the state of this WAL.
`boolean`	`unregisterWALActionsListener(WALActionsListener listener)` Unregisters WALActionsListener
`private static void`	`usage()`

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

- Field Detail
  - LOG
```
private static final org.apache.commons.logging.Log LOG
```
  - DEFAULT_SLOW_SYNC_TIME_MS
```
private static final int DEFAULT_SLOW_SYNC_TIME_MS
```
    See Also:
    Constant Field Values
  - DEFAULT_WAL_SYNC_TIMEOUT_MS
```
private static final int DEFAULT_WAL_SYNC_TIMEOUT_MS
```
    See Also:
    Constant Field Values
  - disruptor
```
private final com.lmax.disruptor.dsl.Disruptor<RingBufferTruck> disruptor
```
    The nexus at which all incoming handlers meet. Does appends and sync with an ordering. Appends and syncs are each put on the ring which means handlers need to smash up against the ring twice (can we make it once only? ... maybe not since time to append is so different from time to sync and sometimes we don't want to sync or we want to async the sync). The ring is where we make sure of our ordering and it is also where we do batching up of handler sync calls.
  - appendExecutor
```
private final ExecutorService appendExecutor
```
    An executorservice that runs the disruptor AppendEventHandler append executor.
  - ringBufferEventHandler
```
private final FSHLog.RingBufferEventHandler ringBufferEventHandler
```
    This fellow is run by the above appendExecutor service but it is all about batching up appends and syncs; it may shutdown without cleaning out the last few appends or syncs. To guard against this, keep a reference to this handler and do explicit close on way out to make sure all flushed out before we exit.
  - cachedSyncFutures
```
private final ThreadLocal<SyncFuture> cachedSyncFutures
```
    Map of SyncFutures owned by Thread objects. Used so we reuse SyncFutures. Thread local is used so JVM can GC the terminated thread for us. See HBASE-21228
  - highestUnsyncedSequence
```
private volatile long highestUnsyncedSequence
```
    The highest known outstanding unsync'd WALEdit sequence number where sequence number is the ring buffer sequence. Maintained by the ring buffer consumer.
  - highestSyncedSequence
```
private final AtomicLong highestSyncedSequence
```
    Updated to the ring buffer sequence of the last successful sync call. This can be less than highestUnsyncedSequence for case where we have an append where a sync has not yet come in for it. Maintained by the syncing threads.
  - fs
```
protected final org.apache.hadoop.fs.FileSystem fs
```
    file system instance
  - fullPathLogDir
```
private final org.apache.hadoop.fs.Path fullPathLogDir
```
    WAL directory, where all WAL files would be placed.
  - fullPathArchiveDir
```
private final org.apache.hadoop.fs.Path fullPathArchiveDir
```
    dir path where old logs are kept.
  - ourFiles
```
private final org.apache.hadoop.fs.PathFilter ourFiles
```
    Matches just those wal files that belong to this wal instance.
  - logFilePrefix
```
private final String logFilePrefix
```
    Prefix of a WAL file, usually the region server name it is hosted on.
  - logFileSuffix
```
private final String logFileSuffix
```
    Suffix included on generated wal file names
  - prefixPathStr
```
private final String prefixPathStr
```
    Prefix used when checking for wal membership.
  - coprocessorHost
```
private final WALCoprocessorHost coprocessorHost
```
  - conf
```
protected final org.apache.hadoop.conf.Configuration conf
```
    conf object
  - listeners
```
private final List<WALActionsListener> listeners
```
    Listeners that are called on WAL events.
  - hdfs_out
```
private org.apache.hadoop.fs.FSDataOutputStream hdfs_out
```
    FSDataOutputStream associated with the current SequenceFile.writer
  - minTolerableReplication
```
private final int minTolerableReplication
```
  - slowSyncNs
```
private final int slowSyncNs
```
  - walSyncTimeout
```
private final long walSyncTimeout
```
  - consecutiveLogRolls
```
private final AtomicInteger consecutiveLogRolls
```
  - lowReplicationRollLimit
```
private final int lowReplicationRollLimit
```
  - lowReplicationRollEnabled
```
private volatile boolean lowReplicationRollEnabled
```
  - sequenceIdAccounting
```
private SequenceIdAccounting sequenceIdAccounting
```
    Class that does accounting of sequenceids in WAL subsystem. Holds oldest outstanding sequence id as yet not flushed as well as the most recent edit sequence id appended to the WAL. Has facility for answering questions such as "Is it safe to GC a WAL?".
  - writer
```
volatile WALProvider.Writer writer
```
    Current log file.
  - closeBarrier
```
private final DrainBarrier closeBarrier
```
    The barrier used to ensure that close() waits for all log rolls and flushes to finish.
  - rollWriterLock
```
private final ReentrantLock rollWriterLock
```
    This lock makes sure only one log roll runs at a time. Should not be taken while any other lock is held. We don't just use synchronized because that results in bogus and tedious findbugs warning when it thinks synchronized controls writer thread safety. It is held when we are actually rolling the log. It is checked when we are looking to see if we should roll the log or not.
  - closed
```
private volatile boolean closed
```
  - shutdown
```
private final AtomicBoolean shutdown
```
  - filenum
```
private final AtomicLong filenum
```
  - numEntries
```
private final AtomicInteger numEntries
```
  - logrollsize
```
private final long logrollsize
```
  - totalLogSize
```
private AtomicLong totalLogSize
```
    The total size of wal
  - maxLogs
```
private final int maxLogs
```
  - closeErrorsTolerated
```
private final int closeErrorsTolerated
```
    Number of log close errors tolerated before we abort
  - closeErrorCount
```
private final AtomicInteger closeErrorCount
```
  - lastTimeCheckLowReplication
```
private volatile long lastTimeCheckLowReplication
```
  - LOG_NAME_COMPARATOR
```
final Comparator<org.apache.hadoop.fs.Path> LOG_NAME_COMPARATOR
```
    WAL Comparator; it compares the timestamp (log filenum), present in the log file name. Throws an IllegalArgumentException if used to compare paths from different wals.
  - byWalRegionSequenceIds
```
private NavigableMap<org.apache.hadoop.fs.Path,Map<byte[],Long>> byWalRegionSequenceIds
```
    Map of WAL log file to the latest sequence ids of all regions it has entries of. The map is sorted by the log file creation timestamp (contained in the log file name).
  - FIXED_OVERHEAD
```
public static final long FIXED_OVERHEAD
```
- Constructor Detail
  - FSHLog
```
public FSHLog(org.apache.hadoop.fs.FileSystem fs,
      org.apache.hadoop.fs.Path root,
      String logDir,
      org.apache.hadoop.conf.Configuration conf)
       throws IOException
```
    Constructor.
    
    Parameters:
    fs - filesystem handle
    root - path for stored and archived wals
    logDir - dir where wals are stored
    conf - configuration to use
    
    Throws:
    
    IOException
  - FSHLog
```
public FSHLog(org.apache.hadoop.fs.FileSystem fs,
      org.apache.hadoop.fs.Path rootDir,
      String logDir,
      String archiveDir,
      org.apache.hadoop.conf.Configuration conf,
      List<WALActionsListener> listeners,
      boolean failIfWALExists,
      String prefix,
      String suffix)
       throws IOException
```
    Create an edit log at the given dir location. You should never have to load an existing log. If there is a log at startup, it should have already been processed and deleted by the time the WAL object is started up.
    
    Parameters:
    fs - filesystem handle
    rootDir - path to where logs and oldlogs
    logDir - dir where wals are stored
    archiveDir - dir where wals are archived
    conf - configuration to use
    listeners - Listeners on WAL events. Listeners passed here will be registered before we do anything else; e.g. the Constructor rollWriter().
    failIfWALExists - If true IOException will be thrown if files related to this wal already exist.
    prefix - should always be hostname and port in distributed env and it will be URL encoded before being used. If prefix is null, "wal" will be used
    suffix - will be url encoded. null is treated as empty. non-empty must start with DefaultWALProvider.WAL_FILE_NAME_DELIMITER
    
    Throws:
    
    IOException
- Method Detail
  - registerWALActionsListener
```
public void registerWALActionsListener(WALActionsListener listener)
```
    Description copied from interface: WAL
    
    Registers WALActionsListener
    
    Specified by:
    
    registerWALActionsListener in interface WAL
  - unregisterWALActionsListener
```
public boolean unregisterWALActionsListener(WALActionsListener listener)
```
    Description copied from interface: WAL
    
    Unregisters WALActionsListener
    
    Specified by:
    
    unregisterWALActionsListener in interface WAL
  - getCoprocessorHost
```
public WALCoprocessorHost getCoprocessorHost()
```
    Specified by:
    
    getCoprocessorHost in interface WAL
    
    Returns:
    Coprocessor host.
  - calculateMaxLogFiles
```
private int calculateMaxLogFiles(float memstoreSizeRatio,
                       long logRollSize)
```
  - getFiles
```
protected org.apache.hadoop.fs.FileStatus[] getFiles()
                                              throws IOException
```
    Get the backing files associated with this WAL.
    
    Returns:
    may be null if there are no files.
    
    Throws:
    
    IOException
  - getOutputStream
```
OutputStream getOutputStream()
```
    Currently, we need to expose the writer's OutputStream to tests so that they can manipulate the default behavior (such as setting the maxRecoveryErrorCount value for example (see TestWALReplay.testReplayEditsWrittenIntoWAL()). This is done using reflection on the underlying HDFS OutputStream. NOTE: This could be removed once Hadoop1 support is removed.
    
    Returns:
    null if underlying stream is not ready.
  - rollWriter
```
public byte[][] rollWriter()
                    throws FailedLogCloseException,
                           IOException
```
    Description copied from interface: WAL
    
    Roll the log writer. That is, start writing log messages to a new file.
    The implementation is synchronized in order to make sure there's one rollWriter running at any given time.
    
    Specified by:
    
    rollWriter in interface WAL
    
    Returns:
    If lots of logs, flush the returned regions so next time through we can clean logs. Returns null if nothing to flush. Names are actual region names as returned by HRegionInfo.getEncodedName()
    
    Throws:
    
    FailedLogCloseException
    
    IOException
  - getNewPath
```
private org.apache.hadoop.fs.Path getNewPath()
                                      throws IOException
```
    retrieve the next path to use for writing. Increments the internal filenum.
    
    Throws:
    
    IOException
  - getOldPath
```
org.apache.hadoop.fs.Path getOldPath()
```
  - tellListenersAboutPreLogRoll
```
private void tellListenersAboutPreLogRoll(org.apache.hadoop.fs.Path oldPath,
                                org.apache.hadoop.fs.Path newPath)
                                   throws IOException
```
    Tell listeners about pre log roll.
    
    Throws:
    
    IOException
  - tellListenersAboutPostLogRoll
```
private void tellListenersAboutPostLogRoll(org.apache.hadoop.fs.Path oldPath,
                                 org.apache.hadoop.fs.Path newPath)
                                    throws IOException
```
    Tell listeners about post log roll.
    
    Throws:
    
    IOException
  - preemptiveSync
```
private void preemptiveSync(ProtobufLogWriter nextWriter)
```
    Run a sync after opening to set up the pipeline.
    
    Parameters:
    nextWriter -
    startTimeNanos -
  - rollWriter
```
public byte[][] rollWriter(boolean force)
                    throws FailedLogCloseException,
                           IOException
```
    Description copied from interface: WAL
    
    Roll the log writer. That is, start writing log messages to a new file.
    The implementation is synchronized in order to make sure there's one rollWriter running at any given time.
    
    Specified by:
    
    rollWriter in interface WAL
    
    Parameters:
    force - If true, force creation of a new writer even if no entries have been written to the current writer
    
    Returns:
    If lots of logs, flush the returned regions so next time through we can clean logs. Returns null if nothing to flush. Names are actual region names as returned by HRegionInfo.getEncodedName()
    
    Throws:
    
    FailedLogCloseException
    
    IOException
  - createWriterInstance
```
protected WALProvider.Writer createWriterInstance(org.apache.hadoop.fs.Path path)
                                           throws IOException
```
    This method allows subclasses to inject different writers without having to extend other methods like rollWriter().
    
    Returns:
    Writer instance
    
    Throws:
    
    IOException
  - cleanOldLogs
```
private void cleanOldLogs()
                   throws IOException
```
    Archive old logs. A WAL is eligible for archiving if all its WALEdits have been flushed.
    
    Throws:
    
    IOException
  - findRegionsToForceFlush
```
byte[][] findRegionsToForceFlush()
                           throws IOException
```
    If the number of un-archived WAL files is greater than maximum allowed, check the first (oldest) WAL file, and returns those regions which should be flushed so that it can be archived.
    
    Returns:
    regions (encodedRegionNames) to flush in order to archive oldest WAL file.
    
    Throws:
    
    IOException
  - afterCreatingZigZagLatch
```
protected void afterCreatingZigZagLatch()
```
    Used to manufacture race condition reliably. For testing only.
    
    See Also:
    beforeWaitOnSafePoint()
  - beforeWaitOnSafePoint
```
protected void beforeWaitOnSafePoint()
```
    See Also:
    afterCreatingZigZagLatch()
  - replaceWriter
```
org.apache.hadoop.fs.Path replaceWriter(org.apache.hadoop.fs.Path oldPath,
                                      org.apache.hadoop.fs.Path newPath,
                                      WALProvider.Writer nextWriter,
                                      org.apache.hadoop.fs.FSDataOutputStream nextHdfsOut)
                                  throws IOException
```
    Cleans up current writer closing it and then puts in place the passed in nextWriter. In the case of creating a new WAL, oldPath will be null. In the case of rolling over from one file to the next, none of the params will be null. In the case of closing out this FSHLog with no further use newPath, nextWriter, and nextHdfsOut will be null.
    
    Parameters:
    oldPath - may be null
    newPath - may be null
    nextWriter - may be null
    nextHdfsOut - may be null
    
    Returns:
    the passed in newPath
    
    Throws:
    
    IOException - if there is a problem flushing or closing the underlying FS
  - getUnflushedEntriesCount
```
long getUnflushedEntriesCount()
```
  - isUnflushedEntries
```
boolean isUnflushedEntries()
```
  - getWALArchivePath
```
public static org.apache.hadoop.fs.Path getWALArchivePath(org.apache.hadoop.fs.Path archiveDir,
                                          org.apache.hadoop.fs.Path p)
```
  - archiveLogFile
```
private void archiveLogFile(org.apache.hadoop.fs.Path p)
                     throws IOException
```
    Throws:
    
    IOException
  - computeFilename
```
protected org.apache.hadoop.fs.Path computeFilename(long filenum)
```
    This is a convenience method that computes a new filename with a given file-number.
    
    Parameters:
    filenum - to use
    
    Returns:
    Path
  - getCurrentFileName
```
public org.apache.hadoop.fs.Path getCurrentFileName()
```
    This is a convenience method that computes a new filename with a given using the current WAL file-number
    
    Returns:
    Path
  - toString
```
public String toString()
```
    Description copied from interface: WAL
    
    Human readable identifying information about the state of this WAL. Implementors are encouraged to include information appropriate for debugging. Consumers are advised not to rely on the details of the returned String; it does not have a defined structure.
    
    Specified by:
    
    toString in interface WAL
    
    Overrides:
    
    toString in class Object
  - getFileNumFromFileName
```
protected long getFileNumFromFileName(org.apache.hadoop.fs.Path fileName)
```
    A log file has a creation timestamp (in ms) in its file name (filenum. This helper method returns the creation timestamp from a given log file. It extracts the timestamp assuming the filename is created with the computeFilename(long filenum) method.
    
    Parameters:
    fileName -
    
    Returns:
    timestamp, as in the log file name.
  - close
```
public void close()
           throws IOException
```
    Description copied from interface: WAL
    
    Caller no longer needs any edits from this WAL. Implementers are free to reclaim underlying resources after this call; i.e. filesystem based WALs can archive or delete files.
    
    Specified by:
    
    close in interface Closeable
    
    Specified by:
    
    close in interface AutoCloseable
    
    Specified by:
    
    close in interface WAL
    
    Throws:
    
    IOException
  - shutdown
```
public void shutdown()
              throws IOException
```
    Description copied from interface: WAL
    
    Stop accepting new writes. If we have unsynced writes still in buffer, sync them. Extant edits are left in place in backing storage to be replayed later.
    
    Specified by:
    
    shutdown in interface WAL
    
    Throws:
    
    IOException
  - append
```
public long append(HTableDescriptor htd,
          HRegionInfo hri,
          WALKey key,
          WALEdit edits,
          boolean inMemstore)
            throws IOException
```
    Description copied from interface: WAL
    
    Append a set of edits to the WAL. The WAL is not flushed/sync'd after this transaction completes BUT on return this edit must have its region edit/sequence id assigned else it messes up our unification of mvcc and sequenceid. On return key will have the region edit/sequence id filled in.
    
    Specified by:
    
    append in interface WAL
    
    Parameters:
    htd - used to give scope for replication TODO refactor out in favor of table name and info
    key - Modified by this call; we add to it this edits region edit/sequence id.
    edits - Edits to append. MAY CONTAIN NO EDITS for case where we want to get an edit sequence id that is after all currently appended edits.
    inMemstore - Always true except for case where we are writing a compaction completion record into the WAL; in this case the entry is just so we can finish an unfinished compaction -- it is not an edit for memstore.
    
    Returns:
    Returns a 'transaction id' and key will have the region edit/sequence id in it.
    
    Throws:
    
    IOException
  - checkLogRoll
```
public void checkLogRoll()
```
    Schedule a log roll if needed.
  - checkLowReplication
```
private boolean checkLowReplication()
```
  - publishSyncOnRingBuffer
```
private SyncFuture publishSyncOnRingBuffer(long sequence)
```
  - getSequenceOnRingBuffer
```
private long getSequenceOnRingBuffer()
```
  - publishSyncOnRingBuffer
```
private SyncFuture publishSyncOnRingBuffer(org.apache.htrace.Span span)
```
  - publishSyncOnRingBuffer
```
private SyncFuture publishSyncOnRingBuffer(long sequence,
                                 org.apache.htrace.Span span)
```
  - publishSyncThenBlockOnCompletion
```
private org.apache.htrace.Span publishSyncThenBlockOnCompletion(org.apache.htrace.Span span)
                                                         throws IOException
```
    Throws:
    
    IOException
  - blockOnSync
```
private org.apache.htrace.Span blockOnSync(SyncFuture syncFuture)
                                    throws IOException
```
    Throws:
    
    IOException
  - convertInterruptedExceptionToIOException
```
private IOException convertInterruptedExceptionToIOException(InterruptedException ie)
```
  - getSyncFuture
```
private SyncFuture getSyncFuture(long sequence,
                       org.apache.htrace.Span span)
```
  - postSync
```
private void postSync(long timeInNanos,
            int handlerSyncs)
```
  - postAppend
```
private long postAppend(WAL.Entry e,
              long elapsedTime)
```
  - getLogReplication
```
int getLogReplication()
```
    This method gets the datanode replication count for the current WAL. If the pipeline isn't started yet or is empty, you will get the default replication factor. Therefore, if this function returns 0, it means you are not properly running with the HDFS-826 patch.
    
    Throws:
    
    InvocationTargetException
    
    IllegalAccessException
    
    IllegalArgumentException
    
    Exception
  - sync
```
public void sync()
          throws IOException
```
    Description copied from interface: WAL
    
    Sync what we have in the WAL.
    
    Specified by:
    
    sync in interface WAL
    
    Throws:
    
    IOException
  - sync
```
public void sync(long txid)
          throws IOException
```
    Description copied from interface: WAL
    
    Sync the WAL if the txId was not already sync'd.
    
    Specified by:
    
    sync in interface WAL
    
    Parameters:
    txid - Transaction id to sync to.
    
    Throws:
    
    IOException
  - requestLogRoll
```
public void requestLogRoll()
```
  - requestLogRoll
```
private void requestLogRoll(boolean tooFewReplicas)
```
  - getNumRolledLogFiles
```
public int getNumRolledLogFiles()
```
    Returns:
    the number of rolled log files
  - getNumLogFiles
```
public int getNumLogFiles()
```
    Returns:
    the number of log files in use
  - getLogFileSize
```
public long getLogFileSize()
```
    Returns:
    the size of log files in use
  - startCacheFlush
```
public Long startCacheFlush(byte[] encodedRegionName,
                   Set<byte[]> families)
```
    Description copied from interface: WAL
    
    WAL keeps track of the sequence numbers that are as yet not flushed im memstores in order to be able to do accounting to figure which WALs can be let go. This method tells WAL that some region is about to flush. The flush can be the whole region or for a column family of the region only.
    Currently, it is expected that the update lock is held for the region; i.e. no concurrent appends while we set up cache flush.
    
    Specified by:
    
    startCacheFlush in interface WAL
    
    families - Families to flush. May be a subset of all families in the region.
    
    Returns:
    Returns HConstants.NO_SEQNUM if we are flushing the whole region OR if we are flushing a subset of all families but there are no edits in those families not being flushed; in other words, this is effectively same as a flush of all of the region though we were passed a subset of regions. Otherwise, it returns the sequence id of the oldest/lowest outstanding edit.
    See Also:
    WAL.completeCacheFlush(byte[]), WAL.abortCacheFlush(byte[])
  - completeCacheFlush
```
public void completeCacheFlush(byte[] encodedRegionName)
```
    Description copied from interface: WAL
    
    Complete the cache flush.
    
    Specified by:
    
    completeCacheFlush in interface WAL
    
    Parameters:
    encodedRegionName - Encoded region name.
    See Also:
    WAL.startCacheFlush(byte[], Set), WAL.abortCacheFlush(byte[])
  - abortCacheFlush
```
public void abortCacheFlush(byte[] encodedRegionName)
```
    Description copied from interface: WAL
    
    Abort a cache flush. Call if the flush fails. Note that the only recovery for an aborted flush currently is a restart of the regionserver so the snapshot content dropped by the failure gets restored to the memstore.
    
    Specified by:
    
    abortCacheFlush in interface WAL
    
    Parameters:
    encodedRegionName - Encoded region name.
  - isLowReplicationRollEnabled
```
boolean isLowReplicationRollEnabled()
```
  - split
```
private static void split(org.apache.hadoop.conf.Configuration conf,
         org.apache.hadoop.fs.Path p)
                   throws IOException
```
    Throws:
    
    IOException
  - getEarliestMemstoreSeqNum
```
public long getEarliestMemstoreSeqNum(byte[] encodedRegionName)
```
    Description copied from interface: WAL
    
    Gets the earliest unflushed sequence id in the memstore for the region.
    
    Specified by:
    
    getEarliestMemstoreSeqNum in interface WAL
    
    Parameters:
    encodedRegionName - The region to get the number for.
    
    Returns:
    The earliest/lowest/oldest sequence id if present, HConstants.NO_SEQNUM if absent.
  - getEarliestMemstoreSeqNum
```
public long getEarliestMemstoreSeqNum(byte[] encodedRegionName,
                             byte[] familyName)
```
    Description copied from interface: WAL
    
    Gets the earliest unflushed sequence id in the memstore for the store.
    
    Specified by:
    
    getEarliestMemstoreSeqNum in interface WAL
    
    Parameters:
    encodedRegionName - The region to get the number for.
    familyName - The family to get the number for.
    
    Returns:
    The earliest/lowest/oldest sequence id if present, HConstants.NO_SEQNUM if absent.
  - atHeadOfRingBufferEventHandlerAppend
```
void atHeadOfRingBufferEventHandlerAppend()
```
    Exposed for testing only. Use to tricks like halt the ring buffer appending.
  - ensureIOException
```
private static IOException ensureIOException(Throwable t)
```
  - usage
```
private static void usage()
```
  - main
```
public static void main(String[] args)
                 throws IOException
```
    Pass one or more log file names and it will either dump out a text version on stdout or split the specified log files.
    
    Parameters:
    args -
    
    Throws:
    
    IOException
  - getPipeLine
```
org.apache.hadoop.hdfs.protocol.DatanodeInfo[] getPipeLine()
```
    This method gets the pipeline for the current WAL.
  - getLastTimeCheckLowReplication
```
public long getLastTimeCheckLowReplication()
```
    Returns:
    last time on checking low replication

Class FSHLog

Failure Semantic

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hadoop.hbase.wal.WAL

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

LOG

DEFAULT_SLOW_SYNC_TIME_MS

DEFAULT_WAL_SYNC_TIMEOUT_MS

disruptor

appendExecutor

ringBufferEventHandler

cachedSyncFutures

highestUnsyncedSequence

highestSyncedSequence

fs

fullPathLogDir

fullPathArchiveDir

ourFiles

logFilePrefix

logFileSuffix

prefixPathStr

coprocessorHost

conf

listeners

hdfs_out

minTolerableReplication

slowSyncNs

walSyncTimeout

consecutiveLogRolls

lowReplicationRollLimit

lowReplicationRollEnabled

sequenceIdAccounting

writer

closeBarrier

rollWriterLock

closed

shutdown

filenum

numEntries

logrollsize

totalLogSize

maxLogs

closeErrorsTolerated

closeErrorCount

lastTimeCheckLowReplication

LOG_NAME_COMPARATOR

byWalRegionSequenceIds

FIXED_OVERHEAD

Constructor Detail

FSHLog

FSHLog

Method Detail

registerWALActionsListener

unregisterWALActionsListener

getCoprocessorHost

calculateMaxLogFiles

getFiles

getOutputStream

rollWriter

getNewPath

getOldPath

tellListenersAboutPreLogRoll

tellListenersAboutPostLogRoll

preemptiveSync

rollWriter

createWriterInstance

cleanOldLogs

findRegionsToForceFlush

afterCreatingZigZagLatch

beforeWaitOnSafePoint

replaceWriter

getUnflushedEntriesCount

isUnflushedEntries

getWALArchivePath

archiveLogFile

computeFilename