Package org.apache.hadoop.hbase.wal

Class WALFactory

java.lang.Object

org.apache.hadoop.hbase.wal.WALFactory

@Private
public class WALFactory
extends Object

Entry point for users of the Write Ahead Log. Acts as the shim between internal use and the particular WALProvider we use to handle wal requests. Configure which provider gets used with the configuration setting "hbase.wal.provider". Available implementations:

• defaultProvider : whatever provider is standard for the hbase version. Currently "asyncfs"
• asyncfs : a provider that will run on top of an implementation of the Hadoop FileSystem interface via an asynchronous client.
• filesystem : a provider that will run on top of an implementation of the Hadoop FileSystem interface via HDFS's synchronous DFSClient.
• multiwal : a provider that will use multiple "filesystem" wal instances per region server.

Alternatively, you may provide a custom implementation of WALProvider by class name.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
(package private) static enum	WALFactory.Providers	Maps between configuration names for providers and implementation classes.

Field Summary

Fields

Modifier and Type	Field	Description
(package private) final Abortable	abortable
private final org.apache.hadoop.conf.Configuration	conf
(package private) static final String	DEFAULT_WAL_PROVIDER
private final ExcludeDatanodeManager	excludeDatanodeManager
(package private) final String	factoryId
private static final org.slf4j.Logger	LOG
static final String	META_WAL_PROVIDER
private final AtomicReference<WALProvider>	metaProvider
private final WALProvider	provider
private static final AtomicReference<WALFactory>	singleton
private static final String	SINGLETON_ID
private final int	timeoutMillis	How long to attempt opening in-recovery wals
static final String	WAL_ENABLED
static final String	WAL_PROVIDER
static final String	WAL_STREAM_READER_CLASS_IMPL	Used in tests for injecting customized stream reader implementation, for example, inject fault when reading, etc.
private final Class<? extends WALStreamReader>	walStreamReaderClass	Configuration-specified WAL Reader used when a custom reader is requested

Constructor Summary

Constructors

Modifier	Constructor	Description
private	WALFactory(org.apache.hadoop.conf.Configuration conf)
	WALFactory(org.apache.hadoop.conf.Configuration conf, String factoryId)
	WALFactory(org.apache.hadoop.conf.Configuration conf, String factoryId, Abortable abortable)

Method Summary

Modifier and Type	Method	Description
void	close()	Shutdown all WALs and clean up any underlying storage.
(package private) WALProvider	createProvider(Class<? extends WALProvider> clazz, String providerId)
WALProvider.Writer	createRecoveredEditsWriter(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path)	Should be package-private, visible for recovery testing.
(package private) static WALProvider.Writer	createRecoveredEditsWriter(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration configuration)	If you already have a WALFactory, you should favor the instance method.
WALStreamReader	createStreamReader(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path)
static WALStreamReader	createStreamReader(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration conf)	Create a one-way stream reader for a given path.
static WALStreamReader	createStreamReader(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration conf, long startPosition)	Create a one-way stream reader for a given path.
WALStreamReader	createStreamReader(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, CancelableProgressable reporter)	Create a one-way stream reader for the WAL.
WALStreamReader	createStreamReader(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, CancelableProgressable reporter, long startPosition)	Create a one-way stream reader for the WAL, and start reading from the given startPosition.
static WALTailingReader	createTailingReader(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration conf, long startPosition)	Create a tailing reader for the given path.
WALProvider.Writer	createWALWriter(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path)	Create a writer for the WAL.
static WALProvider.Writer	createWALWriter(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration configuration)	If you already have a WALFactory, you should favor the instance method.
(package private) WALFactory.Providers	getDefaultProvider()
ExcludeDatanodeManager	getExcludeDatanodeManager()
String	getFactoryId()
static WALFactory	getInstance(org.apache.hadoop.conf.Configuration configuration)
WALProvider	getMetaProvider()	Called when we lazily create a hbase:meta WAL OR from ReplicationSourceManager ahead of creating the first hbase:meta WAL so we can register a listener.
WALProvider	getMetaWALProvider()
(package private) WALProvider	getProvider(String key, String defaultValue, String providerId)	instantiate a provider from a config property.
Class<? extends WALProvider>	getProviderClass(String key, String defaultValue)
WAL	getWAL(RegionInfo region)
WALProvider	getWALProvider()
List<WAL>	getWALs()
void	shutdown()	Tell the underlying WAL providers to shut down, but do not clean up underlying storage.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

WAL_STREAM_READER_CLASS_IMPL

public static final String WAL_STREAM_READER_CLASS_IMPL

Used in tests for injecting customized stream reader implementation, for example, inject fault when reading, etc.

After removing the sequence file based WAL, we always use protobuf based WAL reader, and we will also determine whether the WAL file is encrypted and we should use SecureWALCellCodec to decode by check the header of the WAL file, so we do not need to specify a specical reader to read the WAL file either.

So typically you should not use this config in production.

See Also:

• Constant Field Values

LOG

private static final org.slf4j.Logger LOG

WAL_PROVIDER

public static final String WAL_PROVIDER

See Also:

• Constant Field Values

DEFAULT_WAL_PROVIDER

static final String DEFAULT_WAL_PROVIDER

META_WAL_PROVIDER

public static final String META_WAL_PROVIDER

See Also:

• Constant Field Values

WAL_ENABLED

public static final String WAL_ENABLED

See Also:

• Constant Field Values

factoryId

final String factoryId

abortable

final Abortable abortable

provider

private final WALProvider provider

metaProvider

private final AtomicReference<WALProvider> metaProvider

walStreamReaderClass

private final Class<? extends WALStreamReader> walStreamReaderClass

Configuration-specified WAL Reader used when a custom reader is requested

timeoutMillis

private final int timeoutMillis

How long to attempt opening in-recovery wals

conf

private final org.apache.hadoop.conf.Configuration conf

excludeDatanodeManager

private final ExcludeDatanodeManager excludeDatanodeManager

singleton

private static final AtomicReference<WALFactory> singleton

SINGLETON_ID

private static final String SINGLETON_ID

Constructor Details

WALFactory

private WALFactory(org.apache.hadoop.conf.Configuration conf)

WALFactory

public WALFactory(org.apache.hadoop.conf.Configuration conf,
 String factoryId)
           throws IOException

Throws:

IOException

WALFactory

public WALFactory(org.apache.hadoop.conf.Configuration conf,
 String factoryId,
 Abortable abortable)
           throws IOException

Parameters:

conf - must not be null, will keep a reference to read params in later reader/writer instances.

abortable - the server to abort

Throws:

IOException

Method Details

getDefaultProvider

WALFactory.Providers getDefaultProvider()

getProviderClass

public Class<? extends WALProvider> getProviderClass(String key,
 String defaultValue)

createProvider

WALProvider createProvider(Class<? extends WALProvider> clazz,
 String providerId)
                    throws IOException

Throws:

IOException

getProvider

WALProvider getProvider(String key,
 String defaultValue,
 String providerId)
                 throws IOException

instantiate a provider from a config property. requires conf to have already been set (as well as anything the provider might need to read).

Throws:

IOException

close

public void close()
           throws IOException

Shutdown all WALs and clean up any underlying storage. Use only when you will not need to replay and edits that have gone to any wals from this factory.

Throws:

IOException

shutdown

public void shutdown()
              throws IOException

Tell the underlying WAL providers to shut down, but do not clean up underlying storage. If you are not ending cleanly and will need to replay edits from this factory's wals, use this method if you can as it will try to leave things as tidy as possible.

Throws:

IOException

getWALs

public List<WAL> getWALs()

getMetaProvider

public WALProvider getMetaProvider()
                            throws IOException

Called when we lazily create a hbase:meta WAL OR from ReplicationSourceManager ahead of creating the first hbase:meta WAL so we can register a listener.

Throws:

IOException

See Also:

• getMetaWALProvider()

getWAL

public WAL getWAL(RegionInfo region)
           throws IOException

Parameters:

region - the region which we want to get a WAL for. Could be null.

Throws:

IOException

createStreamReader

public WALStreamReader createStreamReader(org.apache.hadoop.fs.FileSystem fs,
 org.apache.hadoop.fs.Path path)
                                   throws IOException

Throws:

IOException

createStreamReader

public WALStreamReader createStreamReader(org.apache.hadoop.fs.FileSystem fs,
 org.apache.hadoop.fs.Path path,
 CancelableProgressable reporter)
                                   throws IOException

Create a one-way stream reader for the WAL.

Returns:

A WAL reader. Close when done with it.

Throws:

IOException

createStreamReader

public WALStreamReader createStreamReader(org.apache.hadoop.fs.FileSystem fs,
 org.apache.hadoop.fs.Path path,
 CancelableProgressable reporter,
 long startPosition)
                                   throws IOException

Create a one-way stream reader for the WAL, and start reading from the given startPosition.

Returns:

A WAL reader. Close when done with it.

Throws:

IOException

createWALWriter

public WALProvider.Writer createWALWriter(org.apache.hadoop.fs.FileSystem fs,
 org.apache.hadoop.fs.Path path)
                                   throws IOException

Create a writer for the WAL. Uses defaults.

Should be package-private. public only for tests and Compressor

Returns:

A WAL writer. Close when done with it.

Throws:

IOException

createRecoveredEditsWriter

public WALProvider.Writer createRecoveredEditsWriter(org.apache.hadoop.fs.FileSystem fs,
 org.apache.hadoop.fs.Path path)
                                              throws IOException

Should be package-private, visible for recovery testing. Uses defaults.

Returns:

an overwritable writer for recovered edits. caller should close.

Throws:

IOException

getInstance

public static WALFactory getInstance(org.apache.hadoop.conf.Configuration configuration)

createTailingReader

public static WALTailingReader createTailingReader(org.apache.hadoop.fs.FileSystem fs,
 org.apache.hadoop.fs.Path path,
 org.apache.hadoop.conf.Configuration conf,
 long startPosition)
                                            throws IOException

Create a tailing reader for the given path. Mainly used in replication.

Throws:

IOException

createStreamReader

public static WALStreamReader createStreamReader(org.apache.hadoop.fs.FileSystem fs,
 org.apache.hadoop.fs.Path path,
 org.apache.hadoop.conf.Configuration conf)
                                          throws IOException

Create a one-way stream reader for a given path.

Throws:

IOException

createStreamReader

public static WALStreamReader createStreamReader(org.apache.hadoop.fs.FileSystem fs,
 org.apache.hadoop.fs.Path path,
 org.apache.hadoop.conf.Configuration conf,
 long startPosition)
                                          throws IOException

Create a one-way stream reader for a given path.

Throws:

IOException

createRecoveredEditsWriter

static WALProvider.Writer createRecoveredEditsWriter(org.apache.hadoop.fs.FileSystem fs,
 org.apache.hadoop.fs.Path path,
 org.apache.hadoop.conf.Configuration configuration)
                                              throws IOException

If you already have a WALFactory, you should favor the instance method. Uses defaults.

Returns:

a Writer that will overwrite files. Caller must close.

Throws:

IOException

createWALWriter

public static WALProvider.Writer createWALWriter(org.apache.hadoop.fs.FileSystem fs,
 org.apache.hadoop.fs.Path path,
 org.apache.hadoop.conf.Configuration configuration)
                                          throws IOException

If you already have a WALFactory, you should favor the instance method. Uses defaults.

Returns:

a writer that won't overwrite files. Caller must close.

Throws:

IOException

getFactoryId

public String getFactoryId()

getWALProvider

public WALProvider getWALProvider()

getMetaWALProvider

public WALProvider getMetaWALProvider()

Returns:

Current metaProvider... may be null if not yet initialized.

See Also:

• getMetaProvider()

getExcludeDatanodeManager

public ExcludeDatanodeManager getExcludeDatanodeManager()