FSHLog (Apache HBase 1.4.11 API)

java.lang.Object
- org.apache.hadoop.hbase.regionserver.wal.FSHLog

All Implemented Interfaces:

Closeable, AutoCloseable, WAL
```
@InterfaceAudience.Private
public class FSHLog
extends Object
implements WAL
```
Implementation of WAL to go against FileSystem; i.e. keep WALs in HDFS. Only one WAL is ever being written at a time. When a WAL hits a configured maximum size, it is rolled. This is done internal to the implementation.
As data is flushed from the MemStore to other on-disk structures (files sorted by key, hfiles), a WAL becomes obsolete. We can let go of all the log edits/entries for a given HRegion-sequence id. A bunch of work in the below is done keeping account of these region sequence ids -- what is flushed out to hfiles, and what is yet in WAL and in memory only.
It is only practical to delete entire files. Thus, we delete an entire on-disk file F when all of the edits in F have a log-sequence-id that's older (smaller) than the most-recent flush.
To read an WAL, call WALFactory.createReader(org.apache.hadoop.fs.FileSystem, org.apache.hadoop.fs.Path).
Failure Semantic
If an exception on append or sync, roll the WAL because the current WAL is now a lame duck; any more appends or syncs will fail also with the same original exception. If we have made successful appends to the WAL and we then are unable to sync them, our current semantic is to return error to the client that the appends failed but also to abort the current context, usually the hosting server. We need to replay the WALs. TODO: Change this semantic. A roll of WAL may be sufficient as long as we have flagged client that the append failed. TODO: replication may pick up these last edits though they have been marked as failed append (Need to keep our own file lengths, not rely on HDFS).

Nested Class Summary
- Nested classes/interfaces inherited from interface org.apache.hadoop.hbase.wal.WAL
  WAL.Entry, WAL.Reader

Field Summary

Fields
Modifier and Type	Field and Description
`protected org.apache.hadoop.conf.Configuration`	`conf` conf object
`static long`	`FIXED_OVERHEAD`
`protected org.apache.hadoop.fs.FileSystem`	`fs` file system instance

Constructor Summary

Constructors
Constructor and Description
`FSHLog(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path root, String logDir, org.apache.hadoop.conf.Configuration conf)` Constructor.
`FSHLog(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path rootDir, String logDir, String archiveDir, org.apache.hadoop.conf.Configuration conf, List<WALActionsListener> listeners, boolean failIfWALExists, String prefix, String suffix)` Create an edit log at the given `dir` location.

Method Summary

Methods
Modifier and Type	Method and Description
`void`	`abortCacheFlush(byte[] encodedRegionName)` Abort a cache flush.
`protected void`	`afterCreatingZigZagLatch()` Used to manufacture race condition reliably.
`long`	`append(HTableDescriptor htd, HRegionInfo hri, WALKey key, WALEdit edits, boolean inMemstore)` Append a set of edits to the WAL.
`protected void`	`beforeWaitOnSafePoint()`
`void`	`checkLogRoll()` Schedule a log roll if needed.
`void`	`close()` Caller no longer needs any edits from this WAL.
`void`	`completeCacheFlush(byte[] encodedRegionName)` Complete the cache flush.
`protected org.apache.hadoop.fs.Path`	`computeFilename(long filenum)` This is a convenience method that computes a new filename with a given file-number.
`protected WALProvider.Writer`	`createWriterInstance(org.apache.hadoop.fs.Path path)` This method allows subclasses to inject different writers without having to extend other methods like rollWriter().
`WALCoprocessorHost`	`getCoprocessorHost()`
`org.apache.hadoop.fs.Path`	`getCurrentFileName()` This is a convenience method that computes a new filename with a given using the current WAL file-number
`long`	`getEarliestMemstoreSeqNum(byte[] encodedRegionName)` Gets the earliest unflushed sequence id in the memstore for the region.
`long`	`getEarliestMemstoreSeqNum(byte[] encodedRegionName, byte[] familyName)` Gets the earliest unflushed sequence id in the memstore for the store.
`long`	`getFilenum()`
`protected long`	`getFileNumFromFileName(org.apache.hadoop.fs.Path fileName)` A log file has a creation timestamp (in ms) in its file name (`filenum`.
`protected org.apache.hadoop.fs.FileStatus[]`	`getFiles()` Get the backing files associated with this WAL.
`long`	`getLastTimeCheckLowReplication()`
`long`	`getLogFileSize()`
`int`	`getNumLogFiles()`
`int`	`getNumRolledLogFiles()`
`static org.apache.hadoop.fs.Path`	`getWALArchivePath(org.apache.hadoop.fs.Path archiveDir, org.apache.hadoop.fs.Path p)`
`static void`	`main(String[] args)` Pass one or more log file names and it will either dump out a text version on `stdout` or split the specified log files.
`void`	`registerWALActionsListener(WALActionsListener listener)` Registers WALActionsListener
`void`	`requestLogRoll()`
`byte[][]`	`rollWriter()` Roll the log writer.
`byte[][]`	`rollWriter(boolean force)` Roll the log writer.
`void`	`shutdown()` Stop accepting new writes.
`Long`	`startCacheFlush(byte[] encodedRegionName, Set<byte[]> families)` WAL keeps track of the sequence numbers that are as yet not flushed im memstores in order to be able to do accounting to figure which WALs can be let go.
`void`	`sync()` Sync what we have in the WAL.
`void`	`sync(long txid)` Sync the WAL if the txId was not already sync'd.
`String`	`toString()` Human readable identifying information about the state of this WAL.
`boolean`	`unregisterWALActionsListener(WALActionsListener listener)` Unregisters WALActionsListener

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

- Field Detail
  - fs
```
protected final org.apache.hadoop.fs.FileSystem fs
```
    file system instance
  - conf
```
protected final org.apache.hadoop.conf.Configuration conf
```
    conf object
  - FIXED_OVERHEAD
```
public static final long FIXED_OVERHEAD
```
- Constructor Detail
  - FSHLog
```
public FSHLog(org.apache.hadoop.fs.FileSystem fs,
      org.apache.hadoop.fs.Path root,
      String logDir,
      org.apache.hadoop.conf.Configuration conf)
       throws IOException
```
    Constructor.
    
    Parameters:
    fs - filesystem handle
    root - path for stored and archived wals
    logDir - dir where wals are stored
    conf - configuration to use
    
    Throws:
    
    IOException
  - FSHLog
```
public FSHLog(org.apache.hadoop.fs.FileSystem fs,
      org.apache.hadoop.fs.Path rootDir,
      String logDir,
      String archiveDir,
      org.apache.hadoop.conf.Configuration conf,
      List<WALActionsListener> listeners,
      boolean failIfWALExists,
      String prefix,
      String suffix)
       throws IOException
```
    Create an edit log at the given dir location. You should never have to load an existing log. If there is a log at startup, it should have already been processed and deleted by the time the WAL object is started up.
    
    Parameters:
    fs - filesystem handle
    rootDir - path to where logs and oldlogs
    logDir - dir where wals are stored
    archiveDir - dir where wals are archived
    conf - configuration to use
    listeners - Listeners on WAL events. Listeners passed here will be registered before we do anything else; e.g. the Constructor rollWriter().
    failIfWALExists - If true IOException will be thrown if files related to this wal already exist.
    prefix - should always be hostname and port in distributed env and it will be URL encoded before being used. If prefix is null, "wal" will be used
    suffix - will be url encoded. null is treated as empty. non-empty must start with DefaultWALProvider.WAL_FILE_NAME_DELIMITER
    
    Throws:
    
    IOException
- Method Detail
  - registerWALActionsListener
```
public void registerWALActionsListener(WALActionsListener listener)
```
    Description copied from interface: WAL
    
    Registers WALActionsListener
    
    Specified by:
    
    registerWALActionsListener in interface WAL
  - unregisterWALActionsListener
```
public boolean unregisterWALActionsListener(WALActionsListener listener)
```
    Description copied from interface: WAL
    
    Unregisters WALActionsListener
    
    Specified by:
    
    unregisterWALActionsListener in interface WAL
  - getCoprocessorHost
```
public WALCoprocessorHost getCoprocessorHost()
```
    Specified by:
    
    getCoprocessorHost in interface WAL
    
    Returns:
    Coprocessor host.
  - getFiles
```
protected org.apache.hadoop.fs.FileStatus[] getFiles()
                                              throws IOException
```
    Get the backing files associated with this WAL.
    
    Returns:
    may be null if there are no files.
    
    Throws:
    
    IOException
  - rollWriter
```
public byte[][] rollWriter()
                    throws FailedLogCloseException,
                           IOException
```
    Description copied from interface: WAL
    
    Roll the log writer. That is, start writing log messages to a new file.
    The implementation is synchronized in order to make sure there's one rollWriter running at any given time.
    
    Specified by:
    
    rollWriter in interface WAL
    
    Returns:
    If lots of logs, flush the returned regions so next time through we can clean logs. Returns null if nothing to flush. Names are actual region names as returned by HRegionInfo.getEncodedName()
    
    Throws:
    
    FailedLogCloseException
    
    IOException
  - rollWriter
```
public byte[][] rollWriter(boolean force)
                    throws FailedLogCloseException,
                           IOException
```
    Description copied from interface: WAL
    
    Roll the log writer. That is, start writing log messages to a new file.
    The implementation is synchronized in order to make sure there's one rollWriter running at any given time.
    
    Specified by:
    
    rollWriter in interface WAL
    
    Parameters:
    force - If true, force creation of a new writer even if no entries have been written to the current writer
    
    Returns:
    If lots of logs, flush the returned regions so next time through we can clean logs. Returns null if nothing to flush. Names are actual region names as returned by HRegionInfo.getEncodedName()
    
    Throws:
    
    FailedLogCloseException
    
    IOException
  - createWriterInstance
```
protected WALProvider.Writer createWriterInstance(org.apache.hadoop.fs.Path path)
                                           throws IOException
```
    This method allows subclasses to inject different writers without having to extend other methods like rollWriter().
    
    Returns:
    Writer instance
    
    Throws:
    
    IOException
  - afterCreatingZigZagLatch
```
protected void afterCreatingZigZagLatch()
```
    Used to manufacture race condition reliably. For testing only.
    
    See Also:
    beforeWaitOnSafePoint()
  - beforeWaitOnSafePoint
```
protected void beforeWaitOnSafePoint()
```
    See Also:
    afterCreatingZigZagLatch()
  - getWALArchivePath
```
public static org.apache.hadoop.fs.Path getWALArchivePath(org.apache.hadoop.fs.Path archiveDir,
                                          org.apache.hadoop.fs.Path p)
```
  - computeFilename
```
protected org.apache.hadoop.fs.Path computeFilename(long filenum)
```
    This is a convenience method that computes a new filename with a given file-number.
    
    Parameters:
    filenum - to use
    
    Returns:
    Path
  - getCurrentFileName
```
public org.apache.hadoop.fs.Path getCurrentFileName()
```
    This is a convenience method that computes a new filename with a given using the current WAL file-number
    
    Returns:
    Path
  - getFilenum
```
public long getFilenum()
```
    Returns:
    current file number (timestamp)
  - toString
```
public String toString()
```
    Description copied from interface: WAL
    
    Human readable identifying information about the state of this WAL. Implementors are encouraged to include information appropriate for debugging. Consumers are advised not to rely on the details of the returned String; it does not have a defined structure.
    
    Specified by:
    
    toString in interface WAL
    
    Overrides:
    
    toString in class Object
  - getFileNumFromFileName
```
protected long getFileNumFromFileName(org.apache.hadoop.fs.Path fileName)
```
    A log file has a creation timestamp (in ms) in its file name (filenum. This helper method returns the creation timestamp from a given log file. It extracts the timestamp assuming the filename is created with the computeFilename(long filenum) method.
    
    Parameters:
    fileName -
    
    Returns:
    timestamp, as in the log file name.
  - close
```
public void close()
           throws IOException
```
    Description copied from interface: WAL
    
    Caller no longer needs any edits from this WAL. Implementers are free to reclaim underlying resources after this call; i.e. filesystem based WALs can archive or delete files.
    
    Specified by:
    
    close in interface Closeable
    
    Specified by:
    
    close in interface AutoCloseable
    
    Specified by:
    
    close in interface WAL
    
    Throws:
    
    IOException
  - shutdown
```
public void shutdown()
              throws IOException
```
    Description copied from interface: WAL
    
    Stop accepting new writes. If we have unsynced writes still in buffer, sync them. Extant edits are left in place in backing storage to be replayed later.
    
    Specified by:
    
    shutdown in interface WAL
    
    Throws:
    
    IOException
  - append
```
public long append(HTableDescriptor htd,
          HRegionInfo hri,
          WALKey key,
          WALEdit edits,
          boolean inMemstore)
            throws IOException
```
    Description copied from interface: WAL
    
    Append a set of edits to the WAL. The WAL is not flushed/sync'd after this transaction completes BUT on return this edit must have its region edit/sequence id assigned else it messes up our unification of mvcc and sequenceid. On return key will have the region edit/sequence id filled in.
    
    Specified by:
    
    append in interface WAL
    
    Parameters:
    htd - used to give scope for replication TODO refactor out in favor of table name and info
    key - Modified by this call; we add to it this edits region edit/sequence id.
    edits - Edits to append. MAY CONTAIN NO EDITS for case where we want to get an edit sequence id that is after all currently appended edits.
    inMemstore - Always true except for case where we are writing a compaction completion record into the WAL; in this case the entry is just so we can finish an unfinished compaction -- it is not an edit for memstore.
    
    Returns:
    Returns a 'transaction id' and key will have the region edit/sequence id in it.
    
    Throws:
    
    IOException
  - checkLogRoll
```
public void checkLogRoll()
```
    Schedule a log roll if needed.
  - sync
```
public void sync()
          throws IOException
```
    Description copied from interface: WAL
    
    Sync what we have in the WAL.
    
    Specified by:
    
    sync in interface WAL
    
    Throws:
    
    IOException
  - sync
```
public void sync(long txid)
          throws IOException
```
    Description copied from interface: WAL
    
    Sync the WAL if the txId was not already sync'd.
    
    Specified by:
    
    sync in interface WAL
    
    Parameters:
    txid - Transaction id to sync to.
    
    Throws:
    
    IOException
  - requestLogRoll
```
public void requestLogRoll()
```
  - getNumRolledLogFiles
```
public int getNumRolledLogFiles()
```
    Returns:
    the number of rolled log files
  - getNumLogFiles
```
public int getNumLogFiles()
```
    Returns:
    the number of log files in use
  - getLogFileSize
```
public long getLogFileSize()
```
    Returns:
    the size of log files in use
  - startCacheFlush
```
public Long startCacheFlush(byte[] encodedRegionName,
                   Set<byte[]> families)
```
    Description copied from interface: WAL
    
    WAL keeps track of the sequence numbers that are as yet not flushed im memstores in order to be able to do accounting to figure which WALs can be let go. This method tells WAL that some region is about to flush. The flush can be the whole region or for a column family of the region only.
    Currently, it is expected that the update lock is held for the region; i.e. no concurrent appends while we set up cache flush.
    
    Specified by:
    
    startCacheFlush in interface WAL
    
    families - Families to flush. May be a subset of all families in the region.
    
    Returns:
    Returns HConstants.NO_SEQNUM if we are flushing the whole region OR if we are flushing a subset of all families but there are no edits in those families not being flushed; in other words, this is effectively same as a flush of all of the region though we were passed a subset of regions. Otherwise, it returns the sequence id of the oldest/lowest outstanding edit.
    See Also:
    WAL.completeCacheFlush(byte[]), WAL.abortCacheFlush(byte[])
  - completeCacheFlush
```
public void completeCacheFlush(byte[] encodedRegionName)
```
    Description copied from interface: WAL
    
    Complete the cache flush.
    
    Specified by:
    
    completeCacheFlush in interface WAL
    
    Parameters:
    encodedRegionName - Encoded region name.
    See Also:
    WAL.startCacheFlush(byte[], Set), WAL.abortCacheFlush(byte[])
  - abortCacheFlush
```
public void abortCacheFlush(byte[] encodedRegionName)
```
    Description copied from interface: WAL
    
    Abort a cache flush. Call if the flush fails. Note that the only recovery for an aborted flush currently is a restart of the regionserver so the snapshot content dropped by the failure gets restored to the memstore.
    
    Specified by:
    
    abortCacheFlush in interface WAL
    
    Parameters:
    encodedRegionName - Encoded region name.
  - getEarliestMemstoreSeqNum
```
public long getEarliestMemstoreSeqNum(byte[] encodedRegionName)
```
    Description copied from interface: WAL
    
    Gets the earliest unflushed sequence id in the memstore for the region.
    
    Specified by:
    
    getEarliestMemstoreSeqNum in interface WAL
    
    Parameters:
    encodedRegionName - The region to get the number for.
    
    Returns:
    The earliest/lowest/oldest sequence id if present, HConstants.NO_SEQNUM if absent.
  - getEarliestMemstoreSeqNum
```
public long getEarliestMemstoreSeqNum(byte[] encodedRegionName,
                             byte[] familyName)
```
    Description copied from interface: WAL
    
    Gets the earliest unflushed sequence id in the memstore for the store.
    
    Specified by:
    
    getEarliestMemstoreSeqNum in interface WAL
    
    Parameters:
    encodedRegionName - The region to get the number for.
    familyName - The family to get the number for.
    
    Returns:
    The earliest/lowest/oldest sequence id if present, HConstants.NO_SEQNUM if absent.
  - main
```
public static void main(String[] args)
                 throws IOException
```
    Pass one or more log file names and it will either dump out a text version on stdout or split the specified log files.
    
    Parameters:
    args -
    
    Throws:
    
    IOException
  - getLastTimeCheckLowReplication
```
public long getLastTimeCheckLowReplication()
```
    Returns:
    last time on checking low replication

Class FSHLog

Failure Semantic

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hadoop.hbase.wal.WAL

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

fs

conf

FIXED_OVERHEAD

Constructor Detail

FSHLog

FSHLog

Method Detail

registerWALActionsListener

unregisterWALActionsListener

getCoprocessorHost

getFiles

rollWriter

rollWriter

createWriterInstance

afterCreatingZigZagLatch

beforeWaitOnSafePoint

getWALArchivePath

computeFilename

getCurrentFileName

getFilenum

toString

getFileNumFromFileName

close

shutdown

append

checkLogRoll

sync

sync

requestLogRoll

getNumRolledLogFiles

getNumLogFiles

getLogFileSize

startCacheFlush

completeCacheFlush

abortCacheFlush

getEarliestMemstoreSeqNum

getEarliestMemstoreSeqNum

main

getLastTimeCheckLowReplication