AbstractFSWALProvider (Apache HBase 2.5.0 API)

java.lang.Object
- org.apache.hadoop.hbase.wal.AbstractFSWALProvider<T>

All Implemented Interfaces:

WALProvider

Direct Known Subclasses:

AsyncFSWALProvider, FSHLogProvider
```
@InterfaceAudience.Private
 @InterfaceStability.Evolving
public abstract class AbstractFSWALProvider<T extends AbstractFSWAL<?>>
extends Object
implements WALProvider
```
Base class of a WAL Provider that returns a single thread safe WAL that writes to Hadoop FS. By default, this implementation picks a directory in Hadoop FS based on a combination of
- the HBase root directory
- HConstants.HREGION_LOGDIR_NAME
- the given factory's factoryId (usually identifying the regionserver by host:port)
It also uses the providerId to differentiate among files.

Nested Class Summary

Nested Classes
Modifier and Type	Class and Description
`static interface`	`AbstractFSWALProvider.Reader`
`static class`	`AbstractFSWALProvider.WALStartTimeComparator` Comparator used to compare WAL files together based on their start time.

Nested classes/interfaces inherited from interface org.apache.hadoop.hbase.wal.WALProvider
WALProvider.AsyncWriter, WALProvider.Writer, WALProvider.WriterBase

Field Summary

Fields
Modifier and Type	Field and Description
`protected Abortable`	`abortable`
`protected org.apache.hadoop.conf.Configuration`	`conf`
`(package private) static String`	`DEFAULT_PROVIDER_ID`
`static boolean`	`DEFAULT_SEPARATE_OLDLOGDIR`
`protected WALFactory`	`factory`
`protected AtomicBoolean`	`initialized`
`protected List<WALActionsListener>`	`listeners`
`private static org.slf4j.Logger`	`LOG`
`protected String`	`logPrefix`
`static String`	`META_WAL_PROVIDER_ID` The hbase:meta region's WAL filename extension
`private static Pattern`	`pattern` Pattern used to validate a WAL file name see `validateWALFilename(String)` for description.
`protected String`	`providerId`
`static String`	`SEPARATE_OLDLOGDIR` Separate old log into different dir by regionserver name
`static String`	`SPLITTING_EXT` File Extension used while splitting an WAL into regions (HBASE-2312)
`protected T`	`wal`
`static String`	`WAL_FILE_NAME_DELIMITER`
`private ReadWriteLock`	`walCreateLock` We use walCreateLock to prevent wal recreation in different threads, and also prevent getWALs missing the newly created WAL, see HBASE-21503 for more details.

Constructor Summary

Constructors
Constructor and Description

AbstractFSWALProvider()

Constructors
Constructor and Description
`AbstractFSWALProvider()`

Method Summary

All Methods Static Methods Instance Methods Abstract Methods Concrete Methods
Modifier and Type	Method and Description
`void`	`addWALActionsListener(WALActionsListener listener)` Add a `WALActionsListener`.
`void`	`close()` shutdown utstanding WALs and clean up any persisted state.
`protected abstract T`	`createWAL()`
`protected abstract void`	`doInit(org.apache.hadoop.conf.Configuration conf)`
`static long`	`extractFileNumFromWAL(WAL wal)` It returns the file create timestamp from the file name.
`static org.apache.hadoop.fs.Path`	`findArchivedLog(org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration conf)` Find the archived WAL file path if it is not able to locate in WALs dir.
`static org.apache.hadoop.fs.Path`	`getCurrentFileName(WAL wal)` return the current filename from the current wal.
`long`	`getLogFileSize()` iff the given WALFactory is using the DefaultWALProvider for meta and/or non-meta, count the size of files (only rolled).
`static long`	`getLogFileSize(WAL wal)` returns the size of rolled WAL files.
`long`	`getNumLogFiles()` iff the given WALFactory is using the DefaultWALProvider for meta and/or non-meta, count the number of files (rolled and active).
`static int`	`getNumRolledLogFiles(WAL wal)` returns the number of rolled WAL files.
`static ServerName`	`getServerNameFromWALDirectoryName(org.apache.hadoop.conf.Configuration conf, String path)` Pulls a ServerName out of a Path generated according to our layout rules.
`static ServerName`	`getServerNameFromWALDirectoryName(org.apache.hadoop.fs.Path logFile)` This function returns region server name from a log file name which is in one of the following formats: hdfs://<name node>/hbase/.logs/<server name>-splitting/... hdfs://<name node>/hbase/.logs/<server name>/...
`T`	`getWAL(RegionInfo region)`
`static String`	`getWALArchiveDirectoryName(org.apache.hadoop.conf.Configuration conf, String serverName)` Construct the directory name for all old WALs on a given server.
`static String`	`getWALDirectoryName(String serverName)` Construct the directory name for all WALs on a given server.
`static String`	`getWALPrefixFromWALName(String name)` Get prefix of the log from its name, assuming WAL name in format of log_prefix.filenumber.log_suffix
`List<WAL>`	`getWALs()` Returns the List of WALs that are used by this server
`void`	`init(WALFactory factory, org.apache.hadoop.conf.Configuration conf, String providerId, Abortable abortable)` Set up the provider to create wals.
`static boolean`	`isArchivedLogFile(org.apache.hadoop.fs.Path p)`
`static boolean`	`isMetaFile(org.apache.hadoop.fs.Path p)`
`static boolean`	`isMetaFile(String p)` Returns True if String ends in `META_WAL_PROVIDER_ID`
`static WAL.Reader`	`openReader(org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration conf)` Opens WAL reader with retries and additional exception handling
`private static void`	`recoverLease(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path path)`
`(package private) static void`	`requestLogRoll(WAL wal)` request a log roll, but don't actually do it.
`void`	`shutdown()` persist outstanding WALs to storage and stop accepting new appends.
`static boolean`	`validateWALFilename(String filename)` A WAL file name is of the format: <wal-name>`WAL_FILE_NAME_DELIMITER` <file-creation-timestamp>[.meta].

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.hbase.wal.WALProvider
getWALFileLengthProvider

- Field Detail
  - LOG
```
private static final org.slf4j.Logger LOG
```
  - SEPARATE_OLDLOGDIR
```
public static final String SEPARATE_OLDLOGDIR
```
    Separate old log into different dir by regionserver name
    
    See Also:
    
    Constant Field Values
  - DEFAULT_SEPARATE_OLDLOGDIR
```
public static final boolean DEFAULT_SEPARATE_OLDLOGDIR
```
    See Also:
    
    Constant Field Values
  - wal
```
protected volatile T extends AbstractFSWAL<?> wal
```
  - factory
```
protected WALFactory factory
```
  - conf
```
protected org.apache.hadoop.conf.Configuration conf
```
  - listeners
```
protected List<WALActionsListener> listeners
```
  - providerId
```
protected String providerId
```
  - initialized
```
protected AtomicBoolean initialized
```
  - logPrefix
```
protected String logPrefix
```
  - abortable
```
protected Abortable abortable
```
  - walCreateLock
```
private final ReadWriteLock walCreateLock
```
    We use walCreateLock to prevent wal recreation in different threads, and also prevent getWALs missing the newly created WAL, see HBASE-21503 for more details.
  - WAL_FILE_NAME_DELIMITER
```
public static final String WAL_FILE_NAME_DELIMITER
```
    See Also:
    
    Constant Field Values
  - META_WAL_PROVIDER_ID
```
public static final String META_WAL_PROVIDER_ID
```
    The hbase:meta region's WAL filename extension
    
    See Also:
    
    Constant Field Values
  - DEFAULT_PROVIDER_ID
```
static final String DEFAULT_PROVIDER_ID
```
    See Also:
    
    Constant Field Values
  - SPLITTING_EXT
```
public static final String SPLITTING_EXT
```
    File Extension used while splitting an WAL into regions (HBASE-2312)
    
    See Also:
    
    Constant Field Values
  - pattern
```
private static final Pattern pattern
```
    Pattern used to validate a WAL file name see validateWALFilename(String) for description.
- Constructor Detail
  - AbstractFSWALProvider
```
public AbstractFSWALProvider()
```
- Method Detail
  - init
```
public void init(WALFactory factory,
                 org.apache.hadoop.conf.Configuration conf,
                 String providerId,
                 Abortable abortable)
          throws IOException
```
    Description copied from interface: WALProvider
    
    Set up the provider to create wals. will only be called once per instance.
    
    Specified by:
    
    init in interface WALProvider
    
    Parameters:
    
    factory - factory that made us, identity used for FS layout. may not be null
    
    conf - may not be null
    
    providerId - differentiate between providers from one factory, used for FS layout. may be null
    
    Throws:
    
    IOException
  - getWALs
```
public List<WAL> getWALs()
```
    Description copied from interface: WALProvider
    
    Returns the List of WALs that are used by this server
    
    Specified by:
    
    getWALs in interface WALProvider
  - getWAL
```
public T getWAL(RegionInfo region)
         throws IOException
```
    Specified by:
    
    getWAL in interface WALProvider
    
    Parameters:
    
    region - the region which we want to get a WAL for it. Could be null.
    
    Returns:
    
    a WAL for writing entries for the given region.
    
    Throws:
    
    IOException
  - createWAL
```
protected abstract T createWAL()
                        throws IOException
```
    Throws:
    
    IOException
  - doInit
```
protected abstract void doInit(org.apache.hadoop.conf.Configuration conf)
                        throws IOException
```
    Throws:
    
    IOException
  - shutdown
```
public void shutdown()
              throws IOException
```
    Description copied from interface: WALProvider
    
    persist outstanding WALs to storage and stop accepting new appends. This method serves as shorthand for sending a sync to every WAL provided by a given implementation. Those WALs will also stop accepting new writes.
    
    Specified by:
    
    shutdown in interface WALProvider
    
    Throws:
    
    IOException
  - close
```
public void close()
           throws IOException
```
    Description copied from interface: WALProvider
    
    shutdown utstanding WALs and clean up any persisted state. Call this method only when you will not need to replay any of the edits to the WALs from this provider. After this call completes, the underlying resources should have been reclaimed.
    
    Specified by:
    
    close in interface WALProvider
    
    Throws:
    
    IOException
  - getNumLogFiles
```
public long getNumLogFiles()
```
    iff the given WALFactory is using the DefaultWALProvider for meta and/or non-meta, count the number of files (rolled and active). if either of them aren't, count 0 for that provider.
    
    Specified by:
    
    getNumLogFiles in interface WALProvider
  - getLogFileSize
```
public long getLogFileSize()
```
    iff the given WALFactory is using the DefaultWALProvider for meta and/or non-meta, count the size of files (only rolled). if either of them aren't, count 0 for that provider.
    
    Specified by:
    
    getLogFileSize in interface WALProvider
  - getNumRolledLogFiles
```
public static int getNumRolledLogFiles(WAL wal)
```
    returns the number of rolled WAL files.
  - getLogFileSize
```
public static long getLogFileSize(WAL wal)
```
    returns the size of rolled WAL files.
  - getCurrentFileName
```
public static org.apache.hadoop.fs.Path getCurrentFileName(WAL wal)
```
    return the current filename from the current wal.
  - requestLogRoll
```
static void requestLogRoll(WAL wal)
```
    request a log roll, but don't actually do it.
  - extractFileNumFromWAL
```
public static long extractFileNumFromWAL(WAL wal)
```
    It returns the file create timestamp from the file name. For name format see validateWALFilename(String) public until remaining tests move to o.a.h.h.wal
    
    Parameters:
    
    wal - must not be null
    
    Returns:
    
    the file number that is part of the WAL file name
  - validateWALFilename
```
public static boolean validateWALFilename(String filename)
```
    A WAL file name is of the format: <wal-name>WAL_FILE_NAME_DELIMITER <file-creation-timestamp>[.meta]. provider-name is usually made up of a server-name and a provider-id
    
    Parameters:
    
    filename - name of the file to validate
    
    Returns:
    
    true if the filename matches an WAL, false otherwise
  - getWALDirectoryName
```
public static String getWALDirectoryName(String serverName)
```
    Construct the directory name for all WALs on a given server. Dir names currently look like this for WALs: hbase//WALs/kalashnikov.att.net,61634,1486865297088.
    
    Parameters:
    
    serverName - Server name formatted as described in ServerName
    
    Returns:
    
    the relative WAL directory name, e.g. .logs/1.example.org,60030,12345 if serverName passed is 1.example.org,60030,12345
  - getWALArchiveDirectoryName
```
public static String getWALArchiveDirectoryName(org.apache.hadoop.conf.Configuration conf,
                                                String serverName)
```
    Construct the directory name for all old WALs on a given server. The default old WALs dir looks like: hbase/oldWALs. If you config hbase.separate.oldlogdir.by.regionserver to true, it looks like hbase//oldWALs/kalashnikov.att.net,61634,1486865297088.
    
    Parameters:
    
    serverName - Server name formatted as described in ServerName
    
    Returns:
    
    the relative WAL directory name
  - getServerNameFromWALDirectoryName
```
public static ServerName getServerNameFromWALDirectoryName(org.apache.hadoop.conf.Configuration conf,
                                                           String path)
                                                    throws IOException
```
    Pulls a ServerName out of a Path generated according to our layout rules. In the below layouts, this method ignores the format of the logfile component. Current format: [base directory for hbase]/hbase/.logs/ServerName/logfile or [base directory for hbase]/hbase/.logs/ServerName-splitting/logfile Expected to work for individual log files and server-specific directories.
    
    Returns:
    
    null if it's not a log file. Returns the ServerName of the region server that created this log file otherwise.
    
    Throws:
    
    IOException
  - getServerNameFromWALDirectoryName
```
public static ServerName getServerNameFromWALDirectoryName(org.apache.hadoop.fs.Path logFile)
```
    This function returns region server name from a log file name which is in one of the following formats:
    - hdfs://<name node>/hbase/.logs/<server name>-splitting/...
    - hdfs://<name node>/hbase/.logs/<server name>/...
    Returns:
    
    null if the passed in logFile isn't a valid WAL file path
  - isMetaFile
```
public static boolean isMetaFile(org.apache.hadoop.fs.Path p)
```
  - isMetaFile
```
public static boolean isMetaFile(String p)
```
    Returns True if String ends in META_WAL_PROVIDER_ID
  - isArchivedLogFile
```
public static boolean isArchivedLogFile(org.apache.hadoop.fs.Path p)
```
  - findArchivedLog
```
public static org.apache.hadoop.fs.Path findArchivedLog(org.apache.hadoop.fs.Path path,
                                                        org.apache.hadoop.conf.Configuration conf)
                                                 throws IOException
```
    Find the archived WAL file path if it is not able to locate in WALs dir.
    
    Parameters:
    
    path - - active WAL file path
    
    conf - - configuration
    
    Returns:
    
    archived path if exists, null - otherwise
    
    Throws:
    
    IOException - exception
  - openReader
```
public static WAL.Reader openReader(org.apache.hadoop.fs.Path path,
                                    org.apache.hadoop.conf.Configuration conf)
                             throws IOException
```
    Opens WAL reader with retries and additional exception handling
    
    Parameters:
    
    path - path to WAL file
    
    conf - configuration
    
    Returns:
    
    WAL Reader instance
    
    Throws:
    
    IOException
  - recoverLease
```
private static void recoverLease(org.apache.hadoop.conf.Configuration conf,
                                 org.apache.hadoop.fs.Path path)
```
  - addWALActionsListener
```
public void addWALActionsListener(WALActionsListener listener)
```
    Description copied from interface: WALProvider
    
    Add a WALActionsListener.
    Notice that you must call this method before calling WALProvider.getWAL(RegionInfo) as this method will not effect the WAL which has already been created. And as long as we can only it when initialization, it is not thread safe.
    
    Specified by:
    
    addWALActionsListener in interface WALProvider
  - getWALPrefixFromWALName
```
public static String getWALPrefixFromWALName(String name)
```
    Get prefix of the log from its name, assuming WAL name in format of log_prefix.filenumber.log_suffix
    
    Parameters:
    
    name - Name of the WAL to parse
    
    Returns:
    
    prefix of the log
    
    See Also:
    
    AbstractFSWAL.getCurrentFileName()

Class AbstractFSWALProvider<T extends AbstractFSWAL<?>>

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hadoop.hbase.wal.WALProvider

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Methods inherited from interface org.apache.hadoop.hbase.wal.WALProvider

Field Detail

LOG

SEPARATE_OLDLOGDIR

DEFAULT_SEPARATE_OLDLOGDIR

wal

factory

conf

listeners

providerId

initialized

logPrefix

abortable

walCreateLock

WAL_FILE_NAME_DELIMITER

META_WAL_PROVIDER_ID

DEFAULT_PROVIDER_ID

SPLITTING_EXT

pattern

Constructor Detail

AbstractFSWALProvider

Method Detail

init

getWALs

getWAL

createWAL

doInit

shutdown

close

getNumLogFiles

getLogFileSize

getNumRolledLogFiles

getLogFileSize

getCurrentFileName

requestLogRoll

extractFileNumFromWAL

validateWALFilename

getWALDirectoryName

getWALArchiveDirectoryName

getServerNameFromWALDirectoryName

getServerNameFromWALDirectoryName

isMetaFile

isMetaFile

isArchivedLogFile

findArchivedLog

openReader

recoverLease

addWALActionsListener

getWALPrefixFromWALName