org.apache.hadoop.hbase.wal

Class AbstractFSWALProvider<T extends AbstractFSWAL<?>>

java.lang.Object

org.apache.hadoop.hbase.wal.AbstractFSWALProvider<T>

All Implemented Interfaces:

WALProvider

Direct Known Subclasses:

AsyncFSWALProvider, FSHLogProvider

@InterfaceAudience.Private
 @InterfaceStability.Evolving
public abstract class AbstractFSWALProvider<T extends AbstractFSWAL<?>>
extends Object
implements WALProvider

Base class of a WAL Provider that returns a single thread safe WAL that writes to Hadoop FS. By default, this implementation picks a directory in Hadoop FS based on a combination of

• the HBase root directory
• HConstants.HREGION_LOGDIR_NAME
• the given factory's factoryId (usually identifying the regionserver by host:port)

It also uses the providerId to differentiate among files.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	AbstractFSWALProvider.Reader

Nested classes/interfaces inherited from interface org.apache.hadoop.hbase.wal.WALProvider

WALProvider.AsyncWriter, WALProvider.Writer, WALProvider.WriterBase

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.hadoop.conf.Configuration	conf
(package private) static String	DEFAULT_PROVIDER_ID
static boolean	DEFAULT_SEPARATE_OLDLOGDIR
protected WALFactory	factory
protected AtomicBoolean	initialized
protected List<WALActionsListener>	listeners
private static org.slf4j.Logger	LOG
protected String	logPrefix
static String	META_WAL_PROVIDER_ID The hbase:meta region's WAL filename extension
private static Pattern	pattern Pattern used to validate a WAL file name see validateWALFilename(String) for description.
protected String	providerId
static String	SEPARATE_OLDLOGDIR Separate old log into different dir by regionserver name
static String	SPLITTING_EXT File Extension used while splitting an WAL into regions (HBASE-2312)
protected T	wal
static String	WAL_FILE_NAME_DELIMITER
private ReadWriteLock	walCreateLock We use walCreateLock to prevent wal recreation in different threads, and also prevent getWALs missing the newly created WAL, see HBASE-21503 for more details.

Constructor Summary

Constructors

Constructor and Description
AbstractFSWALProvider()

Method Summary

Modifier and Type	Method and Description
void	addWALActionsListener(WALActionsListener listener) Add a WALActionsListener.
void	close() shutdown utstanding WALs and clean up any persisted state.
protected abstract T	createWAL()
protected abstract void	doInit(org.apache.hadoop.conf.Configuration conf)
static long	extractFileNumFromWAL(WAL wal) It returns the file create timestamp from the file name.
static org.apache.hadoop.fs.Path	getArchivedLogPath(org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration conf) Get the archived WAL file path
static org.apache.hadoop.fs.Path	getCurrentFileName(WAL wal) return the current filename from the current wal.
long	getLogFileSize() iff the given WALFactory is using the DefaultWALProvider for meta and/or non-meta, count the size of files (only rolled).
static long	getLogFileSize(WAL wal) returns the size of rolled WAL files.
long	getNumLogFiles() iff the given WALFactory is using the DefaultWALProvider for meta and/or non-meta, count the number of files (rolled and active).
static int	getNumRolledLogFiles(WAL wal) returns the number of rolled WAL files.
static ServerName	getServerNameFromWALDirectoryName(org.apache.hadoop.conf.Configuration conf, String path) Pulls a ServerName out of a Path generated according to our layout rules.
static ServerName	getServerNameFromWALDirectoryName(org.apache.hadoop.fs.Path logFile) This function returns region server name from a log file name which is in one of the following formats: hdfs://<name node>/hbase/.logs/<server name>-splitting/... hdfs://<name node>/hbase/.logs/<server name>/...
T	getWAL(RegionInfo region)
static String	getWALArchiveDirectoryName(org.apache.hadoop.conf.Configuration conf, String serverName) Construct the directory name for all old WALs on a given server.
static String	getWALDirectoryName(String serverName) Construct the directory name for all WALs on a given server.
static String	getWALPrefixFromWALName(String name) Get prefix of the log from its name, assuming WAL name in format of log_prefix.filenumber.log_suffix
List<WAL>	getWALs()
void	init(WALFactory factory, org.apache.hadoop.conf.Configuration conf, String providerId) Set up the provider to create wals.
static boolean	isArchivedLogFile(org.apache.hadoop.fs.Path p)
static boolean	isMetaFile(org.apache.hadoop.fs.Path p)
static boolean	isMetaFile(String p)
static WAL.Reader	openReader(org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration conf) Opens WAL reader with retries and additional exception handling
private static void	recoverLease(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path path)
(package private) static void	requestLogRoll(WAL wal) request a log roll, but don't actually do it.
void	shutdown() persist outstanding WALs to storage and stop accepting new appends.
static boolean	validateWALFilename(String filename) A WAL file name is of the format: <wal-name>WAL_FILE_NAME_DELIMITER <file-creation-timestamp>[.meta].

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.hbase.wal.WALProvider

getWALFileLengthProvider

Field Detail

LOG

private static final org.slf4j.Logger LOG

SEPARATE_OLDLOGDIR

public static final String SEPARATE_OLDLOGDIR

Separate old log into different dir by regionserver name

See Also:

Constant Field Values

DEFAULT_SEPARATE_OLDLOGDIR

public static final boolean DEFAULT_SEPARATE_OLDLOGDIR

See Also:

Constant Field Values

wal

protected volatile T extends AbstractFSWAL<?> wal

factory

protected WALFactory factory

conf

protected org.apache.hadoop.conf.Configuration conf

listeners

protected List<WALActionsListener> listeners

providerId

protected String providerId

initialized

protected AtomicBoolean initialized

logPrefix

protected String logPrefix

walCreateLock

private final ReadWriteLock walCreateLock

We use walCreateLock to prevent wal recreation in different threads, and also prevent getWALs missing the newly created WAL, see HBASE-21503 for more details.

WAL_FILE_NAME_DELIMITER

public static final String WAL_FILE_NAME_DELIMITER

See Also:

Constant Field Values

META_WAL_PROVIDER_ID

public static final String META_WAL_PROVIDER_ID

The hbase:meta region's WAL filename extension

See Also:

Constant Field Values

DEFAULT_PROVIDER_ID

static final String DEFAULT_PROVIDER_ID

See Also:

Constant Field Values

SPLITTING_EXT

public static final String SPLITTING_EXT

File Extension used while splitting an WAL into regions (HBASE-2312)

See Also:

Constant Field Values

pattern

private static final Pattern pattern

Pattern used to validate a WAL file name see validateWALFilename(String) for description.

Constructor Detail

AbstractFSWALProvider

public AbstractFSWALProvider()

Method Detail

init

public void init(WALFactory factory,
                 org.apache.hadoop.conf.Configuration conf,
                 String providerId)
          throws IOException

Description copied from interface: WALProvider

Set up the provider to create wals. will only be called once per instance.

Specified by:

init in interface WALProvider

Parameters:

factory - factory that made us, identity used for FS layout. may not be null

conf - may not be null

providerId - differentiate between providers from one factory, used for FS layout. may be null

Throws:

IOException

getWALs

public List<WAL> getWALs()

Specified by:

getWALs in interface WALProvider

Returns:

the List of WALs that are used by this server

getWAL

public T getWAL(RegionInfo region)
         throws IOException

Specified by:

getWAL in interface WALProvider

Parameters:

region - the region which we want to get a WAL for it. Could be null.

Returns:

a WAL for writing entries for the given region.

Throws:

IOException

createWAL

protected abstract T createWAL()
                        throws IOException

Throws:

IOException

doInit

protected abstract void doInit(org.apache.hadoop.conf.Configuration conf)
                        throws IOException

Throws:

IOException

shutdown

public void shutdown()
              throws IOException

Description copied from interface: WALProvider

persist outstanding WALs to storage and stop accepting new appends. This method serves as shorthand for sending a sync to every WAL provided by a given implementation. Those WALs will also stop accepting new writes.

Specified by:

shutdown in interface WALProvider

Throws:

IOException

close

public void close()
           throws IOException

Description copied from interface: WALProvider

shutdown utstanding WALs and clean up any persisted state. Call this method only when you will not need to replay any of the edits to the WALs from this provider. After this call completes, the underlying resources should have been reclaimed.

Specified by:

close in interface WALProvider

Throws:

IOException

getNumLogFiles

public long getNumLogFiles()

iff the given WALFactory is using the DefaultWALProvider for meta and/or non-meta, count the number of files (rolled and active). if either of them aren't, count 0 for that provider.

Specified by:

getNumLogFiles in interface WALProvider

getLogFileSize

public long getLogFileSize()

iff the given WALFactory is using the DefaultWALProvider for meta and/or non-meta, count the size of files (only rolled). if either of them aren't, count 0 for that provider.

Specified by:

getLogFileSize in interface WALProvider

getNumRolledLogFiles

public static int getNumRolledLogFiles(WAL wal)

returns the number of rolled WAL files.

getLogFileSize

public static long getLogFileSize(WAL wal)

returns the size of rolled WAL files.

getCurrentFileName

public static org.apache.hadoop.fs.Path getCurrentFileName(WAL wal)

return the current filename from the current wal.

requestLogRoll

static void requestLogRoll(WAL wal)

request a log roll, but don't actually do it.

extractFileNumFromWAL

public static long extractFileNumFromWAL(WAL wal)

It returns the file create timestamp from the file name. For name format see validateWALFilename(String) public until remaining tests move to o.a.h.h.wal

Parameters:

wal - must not be null

Returns:

the file number that is part of the WAL file name

validateWALFilename

public static boolean validateWALFilename(String filename)

A WAL file name is of the format: <wal-name>WAL_FILE_NAME_DELIMITER <file-creation-timestamp>[.meta]. provider-name is usually made up of a server-name and a provider-id

Parameters:

filename - name of the file to validate

Returns:

true if the filename matches an WAL, false otherwise

getWALDirectoryName

public static String getWALDirectoryName(String serverName)

Construct the directory name for all WALs on a given server. Dir names currently look like this for WALs: hbase//WALs/kalashnikov.att.net,61634,1486865297088.

Parameters:

serverName - Server name formatted as described in ServerName

Returns:

the relative WAL directory name, e.g. .logs/1.example.org,60030,12345 if serverName passed is 1.example.org,60030,12345

getWALArchiveDirectoryName

public static String getWALArchiveDirectoryName(org.apache.hadoop.conf.Configuration conf,
                                                String serverName)

Construct the directory name for all old WALs on a given server. The default old WALs dir looks like: hbase/oldWALs. If you config hbase.separate.oldlogdir.by.regionserver to true, it looks like hbase//oldWALs/kalashnikov.att.net,61634,1486865297088.

Parameters:

conf -

serverName - Server name formatted as described in ServerName

Returns:

the relative WAL directory name

getServerNameFromWALDirectoryName

public static ServerName getServerNameFromWALDirectoryName(org.apache.hadoop.conf.Configuration conf,
                                                           String path)
                                                    throws IOException

Pulls a ServerName out of a Path generated according to our layout rules. In the below layouts, this method ignores the format of the logfile component. Current format: [base directory for hbase]/hbase/.logs/ServerName/logfile or [base directory for hbase]/hbase/.logs/ServerName-splitting/logfile Expected to work for individual log files and server-specific directories.

Returns:

null if it's not a log file. Returns the ServerName of the region server that created this log file otherwise.

Throws:

IOException

getServerNameFromWALDirectoryName

public static ServerName getServerNameFromWALDirectoryName(org.apache.hadoop.fs.Path logFile)

This function returns region server name from a log file name which is in one of the following formats:

• hdfs://<name node>/hbase/.logs/<server name>-splitting/...
• hdfs://<name node>/hbase/.logs/<server name>/...

Returns:

null if the passed in logFile isn't a valid WAL file path

isMetaFile

public static boolean isMetaFile(org.apache.hadoop.fs.Path p)

isMetaFile

public static boolean isMetaFile(String p)

isArchivedLogFile

public static boolean isArchivedLogFile(org.apache.hadoop.fs.Path p)

getArchivedLogPath

public static org.apache.hadoop.fs.Path getArchivedLogPath(org.apache.hadoop.fs.Path path,
                                                           org.apache.hadoop.conf.Configuration conf)
                                                    throws IOException

Get the archived WAL file path

Parameters:

path - - active WAL file path

conf - - configuration

Returns:

archived path if exists, path - otherwise

Throws:

IOException - exception

openReader

public static WAL.Reader openReader(org.apache.hadoop.fs.Path path,
                                    org.apache.hadoop.conf.Configuration conf)
                             throws IOException

Opens WAL reader with retries and additional exception handling

Parameters:

path - path to WAL file

conf - configuration

Returns:

WAL Reader instance

Throws:

IOException

recoverLease

private static void recoverLease(org.apache.hadoop.conf.Configuration conf,
                                 org.apache.hadoop.fs.Path path)

addWALActionsListener

public void addWALActionsListener(WALActionsListener listener)

Description copied from interface: WALProvider

Add a WALActionsListener.

Notice that you must call this method before calling WALProvider.getWAL(RegionInfo) as this method will not effect the WAL which has already been created. And as long as we can only it when initialization, it is not thread safe.

Specified by:

addWALActionsListener in interface WALProvider

getWALPrefixFromWALName

public static String getWALPrefixFromWALName(String name)

Get prefix of the log from its name, assuming WAL name in format of log_prefix.filenumber.log_suffix

Parameters:

name - Name of the WAL to parse

Returns:

prefix of the log

See Also:

AbstractFSWAL.getCurrentFileName()