Package org.apache.hadoop.hbase.regionserver.wal

Class FSHLog

java.lang.Object

org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL<WALProvider.Writer>

org.apache.hadoop.hbase.regionserver.wal.FSHLog

All Implemented Interfaces:

Closeable, AutoCloseable, WALFileLengthProvider, WAL

@Private
public class FSHLog
extends AbstractFSWAL<WALProvider.Writer>

The original implementation of FSWAL.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
(package private) static class	FSHLog.SyncRequest
private class	FSHLog.SyncRunner	Thread to runs the hdfs sync call.

Nested classes/interfaces inherited from interface org.apache.hadoop.hbase.wal.WAL

WAL.Entry

Field Summary

Fields

Modifier and Type	Field	Description
private final AtomicInteger	consecutiveLogRolls
private static final int	DEFAULT_FSHLOG_WAIT_ON_SHUTDOWN_IN_SECONDS
private static final int	DEFAULT_LOW_REPLICATION_ROLL_LIMIT
private static final int	DEFAULT_MAX_BATCH_COUNT
private static final int	DEFAULT_SYNCER_COUNT
static final long	FIXED_OVERHEAD
private static final String	FSHLOG_WAIT_ON_SHUTDOWN_IN_SECONDS
private org.apache.hadoop.fs.FSDataOutputStream	hdfs_out	FSDataOutputStream associated with the current SequenceFile.writer
private static final org.slf4j.Logger	LOG
private static final String	LOW_REPLICATION_ROLL_LIMIT
private boolean	lowReplicationRollEnabled
private final int	lowReplicationRollLimit
private static final String	MAX_BATCH_COUNT
private final int	maxSyncRequestCount
private final int	minTolerableReplication
private static final String	SYNCER_COUNT
private final int	syncerCount
private int	syncRunnerIndex	Which syncrunner to use next.
private FSHLog.SyncRunner[]	syncRunners
private static final String	TOLERABLE_LOW_REPLICATION
private static final IOException	WAL_CLOSE_EXCEPTION
private static final IOException	WITER_BROKEN_EXCEPTION
private static final IOException	WITER_REPLACED_EXCEPTION

Fields inherited from class org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL

abortable, blocksize, closed, closeExecutor, conf, consumeExecutor, consumer, coprocessorHost, DEFAULT_ROLL_ON_SYNC_TIME_MS, DEFAULT_SLOW_SYNC_ROLL_INTERVAL_MS, DEFAULT_SLOW_SYNC_ROLL_THRESHOLD, DEFAULT_SLOW_SYNC_TIME_MS, DEFAULT_WAL_BATCH_SIZE, DEFAULT_WAL_SHUTDOWN_WAIT_TIMEOUT_MS, DEFAULT_WAL_SYNC_TIMEOUT_MS, filenum, fs, hasConsumerTask, highestProcessedAppendTxid, highestSyncedTxid, highestUnsyncedTxid, implClassName, inflightWALClosures, listeners, LOG_NAME_COMPARATOR, logrollsize, MAX_LOGS, maxLogs, numEntries, ourFiles, prefixPathStr, RING_BUFFER_SLOT_COUNT, ROLL_ON_SYNC_TIME_MS, rollOnSyncNs, rollRequested, rollWriterLock, sequenceIdAccounting, shouldShutDownConsumeExecutorWhenClose, shutdown, SLOW_SYNC_ROLL_INTERVAL_MS, SLOW_SYNC_ROLL_THRESHOLD, SLOW_SYNC_TIME_MS, slowSyncCheckInterval, slowSyncCount, slowSyncNs, slowSyncRollThreshold, syncFutureCache, syncFutures, totalLogSize, toWriteAppends, unackedAppends, useHsync, WAL_AVOID_LOCAL_WRITES_DEFAULT, WAL_AVOID_LOCAL_WRITES_KEY, WAL_BATCH_SIZE, WAL_ROLL_MULTIPLIER, WAL_SHUTDOWN_WAIT_TIMEOUT_MS, WAL_SYNC_TIMEOUT_MS, walArchiveDir, walDir, walFile2Props, walFilePrefix, walFileSuffix, walShutdownTimeout, writer

Constructor Summary

Constructors

Constructor	Description
FSHLog(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path rootDir, String logDir, String archiveDir, org.apache.hadoop.conf.Configuration conf, List<WALActionsListener> listeners, boolean failIfWALExists, String prefix, String suffix)
FSHLog(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path root, String logDir, org.apache.hadoop.conf.Configuration conf)	Constructor.
FSHLog(org.apache.hadoop.fs.FileSystem fs, Abortable abortable, org.apache.hadoop.fs.Path rootDir, String logDir, String archiveDir, org.apache.hadoop.conf.Configuration conf, List<WALActionsListener> listeners, boolean failIfWALExists, String prefix, String suffix, org.apache.hadoop.fs.FileSystem remoteFs, org.apache.hadoop.fs.Path remoteWALDir)	Create an edit log at the given dir location.
FSHLog(org.apache.hadoop.fs.FileSystem fs, Abortable abortable, org.apache.hadoop.fs.Path root, String logDir, org.apache.hadoop.conf.Configuration conf)

Method Summary

Modifier and Type	Method	Description
protected void	checkSlowSyncCount()
protected WALProvider.Writer	createCombinedWriter(WALProvider.Writer localWriter, WALProvider.Writer remoteWriter)
private void	createSyncRunnersAndStart()
protected WALProvider.Writer	createWriterInstance(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path)	This method allows subclasses to inject different writers without having to extend other methods like rollWriter().
protected void	doAppend(WALProvider.Writer writer, FSWALEntry entry)
protected boolean	doCheckLogLowReplication()	Returns true if number of replicas for the WAL is lower than threshold
protected void	doCleanUpResources()
protected CompletableFuture<Long>	doWriterSync(WALProvider.Writer writer, boolean shouldUseHSync, long txidWhenSync)
(package private) int	getLogReplication()	This method gets the datanode replication count for the current WAL.
(package private) OutputStream	getOutputStream()	Currently, we need to expose the writer's OutputStream to tests so that they can manipulate the default behavior (such as setting the maxRecoveryErrorCount value).
(package private) org.apache.hadoop.hdfs.protocol.DatanodeInfo[]	getPipeline()	This method gets the pipeline for the current WAL.
void	init()	Used to initialize the WAL.
(package private) boolean	isLowReplicationRollEnabled()
private void	offerSyncRequest(FSHLog.SyncRequest syncRequest)
protected void	onWriterReplaced(WALProvider.Writer nextWriter)
private void	preemptiveSync(ProtobufLogWriter nextWriter)	Run a sync after opening to set up the pipeline.
private void	shutDownSyncRunners()

Methods inherited from class org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL

abortCacheFlush, append, appendData, appendEntry, appendMarker, archive, archiveLogFile, atHeadOfRingBufferEventHandlerAppend, blockOnSync, checkLogLowReplication, close, closeWriter, completeCacheFlush, computeFilename, createSingleThreadPoolConsumeExecutor, doCheckSlowSync, doReplaceWriter, doShutdown, doSync, doSync, findRegionsToForceFlush, getCoprocessorHost, getCurrentFileName, getEarliestMemStoreSeqNum, getFilenum, getFileNumFromFileName, getFiles, getInflightWALCloseCount, getLogFileSize, getLogFileSizeIfBeingWritten, getNumLogFiles, getNumRolledLogFiles, getOldPath, getPreallocatedEventCount, getSequenceIdAccounting, getSyncedTxid, getSyncFuture, getUnflushedEntriesCount, getWALArchivePath, getWriter, isLogRollRequested, isUnflushedEntries, isWriterBroken, logRollAndSetupWalProps, main, markFutureDoneAndOffer, postSync, registerWALActionsListener, replaceWriter, requestLogRoll, requestLogRoll, rollWriter, rollWriter, setWaitOnShutdownInSeconds, shutdown, skipRemoteWAL, stampSequenceIdAndPublishToRingBuffer, startCacheFlush, startCacheFlush, sync, sync, sync, sync, toString, unregisterWALActionsListener, updateStore, waitForSafePoint

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Details

LOG

private static final org.slf4j.Logger LOG

TOLERABLE_LOW_REPLICATION

private static final String TOLERABLE_LOW_REPLICATION

See Also:

• Constant Field Values

LOW_REPLICATION_ROLL_LIMIT

private static final String LOW_REPLICATION_ROLL_LIMIT

See Also:

• Constant Field Values

DEFAULT_LOW_REPLICATION_ROLL_LIMIT

private static final int DEFAULT_LOW_REPLICATION_ROLL_LIMIT

See Also:

• Constant Field Values

SYNCER_COUNT

private static final String SYNCER_COUNT

See Also:

• Constant Field Values

DEFAULT_SYNCER_COUNT

private static final int DEFAULT_SYNCER_COUNT

See Also:

• Constant Field Values

MAX_BATCH_COUNT

private static final String MAX_BATCH_COUNT

See Also:

• Constant Field Values

DEFAULT_MAX_BATCH_COUNT

private static final int DEFAULT_MAX_BATCH_COUNT

See Also:

• Constant Field Values

FSHLOG_WAIT_ON_SHUTDOWN_IN_SECONDS

private static final String FSHLOG_WAIT_ON_SHUTDOWN_IN_SECONDS

See Also:

• Constant Field Values

DEFAULT_FSHLOG_WAIT_ON_SHUTDOWN_IN_SECONDS

private static final int DEFAULT_FSHLOG_WAIT_ON_SHUTDOWN_IN_SECONDS

See Also:

• Constant Field Values

WITER_REPLACED_EXCEPTION

private static final IOException WITER_REPLACED_EXCEPTION

WITER_BROKEN_EXCEPTION

private static final IOException WITER_BROKEN_EXCEPTION

WAL_CLOSE_EXCEPTION

private static final IOException WAL_CLOSE_EXCEPTION

hdfs_out

private org.apache.hadoop.fs.FSDataOutputStream hdfs_out

FSDataOutputStream associated with the current SequenceFile.writer

minTolerableReplication

private final int minTolerableReplication

consecutiveLogRolls

private final AtomicInteger consecutiveLogRolls

lowReplicationRollLimit

private final int lowReplicationRollLimit

lowReplicationRollEnabled

private volatile boolean lowReplicationRollEnabled

syncerCount

private final int syncerCount

maxSyncRequestCount

private final int maxSyncRequestCount

syncRunnerIndex

private int syncRunnerIndex

Which syncrunner to use next.

syncRunners

private FSHLog.SyncRunner[] syncRunners

FIXED_OVERHEAD

public static final long FIXED_OVERHEAD

Constructor Details

FSHLog

public FSHLog(org.apache.hadoop.fs.FileSystem fs,
 org.apache.hadoop.fs.Path root,
 String logDir,
 org.apache.hadoop.conf.Configuration conf)
       throws IOException

Constructor.

Parameters:

fs - filesystem handle

root - path for stored and archived wals

logDir - dir where wals are stored

conf - configuration to use

Throws:

IOException

FSHLog

public FSHLog(org.apache.hadoop.fs.FileSystem fs,
 Abortable abortable,
 org.apache.hadoop.fs.Path root,
 String logDir,
 org.apache.hadoop.conf.Configuration conf)
       throws IOException

Throws:

IOException

FSHLog

public FSHLog(org.apache.hadoop.fs.FileSystem fs,
 org.apache.hadoop.fs.Path rootDir,
 String logDir,
 String archiveDir,
 org.apache.hadoop.conf.Configuration conf,
 List<WALActionsListener> listeners,
 boolean failIfWALExists,
 String prefix,
 String suffix)
       throws IOException

Throws:

IOException

FSHLog

public FSHLog(org.apache.hadoop.fs.FileSystem fs,
 Abortable abortable,
 org.apache.hadoop.fs.Path rootDir,
 String logDir,
 String archiveDir,
 org.apache.hadoop.conf.Configuration conf,
 List<WALActionsListener> listeners,
 boolean failIfWALExists,
 String prefix,
 String suffix,
 org.apache.hadoop.fs.FileSystem remoteFs,
 org.apache.hadoop.fs.Path remoteWALDir)
       throws IOException

Create an edit log at the given dir location. You should never have to load an existing log. If there is a log at startup, it should have already been processed and deleted by the time the WAL object is started up.

Parameters:

fs - filesystem handle

abortable - Abortable - the server here

rootDir - path to where logs and oldlogs

logDir - dir where wals are stored

archiveDir - dir where wals are archived

conf - configuration to use

listeners - Listeners on WAL events. Listeners passed here will be registered before we do anything else; e.g. the Constructor AbstractFSWAL.rollWriter().

failIfWALExists - If true IOException will be thrown if files related to this wal already exist.

prefix - should always be hostname and port in distributed env and it will be URL encoded before being used. If prefix is null, "wal" will be used

suffix - will be url encoded. null is treated as empty. non-empty must start with AbstractFSWALProvider.WAL_FILE_NAME_DELIMITER

Throws:

IOException

Method Details

init

public void init()
          throws IOException

Description copied from class: AbstractFSWAL

Used to initialize the WAL. Usually just call rollWriter to create the first log writer.

Specified by:

init in interface WAL

Overrides:

init in class AbstractFSWAL<WALProvider.Writer>

Throws:

IOException

createSyncRunnersAndStart

private void createSyncRunnersAndStart()

getOutputStream

OutputStream getOutputStream()

Currently, we need to expose the writer's OutputStream to tests so that they can manipulate the default behavior (such as setting the maxRecoveryErrorCount value). This is done using reflection on the underlying HDFS OutputStream. NOTE: This could be removed once Hadoop1 support is removed.

Returns:

null if underlying stream is not ready.

preemptiveSync

private void preemptiveSync(ProtobufLogWriter nextWriter)

Run a sync after opening to set up the pipeline.

createWriterInstance

protected WALProvider.Writer createWriterInstance(org.apache.hadoop.fs.FileSystem fs,
 org.apache.hadoop.fs.Path path)
                                           throws IOException

This method allows subclasses to inject different writers without having to extend other methods like rollWriter().

Specified by:

createWriterInstance in class AbstractFSWAL<WALProvider.Writer>

Returns:

Writer instance

Throws:

IOException

doAppend

protected void doAppend(WALProvider.Writer writer,
 FSWALEntry entry)
                 throws IOException

Specified by:

doAppend in class AbstractFSWAL<WALProvider.Writer>

Throws:

IOException

onWriterReplaced

protected void onWriterReplaced(WALProvider.Writer nextWriter)

Specified by:

onWriterReplaced in class AbstractFSWAL<WALProvider.Writer>

doCleanUpResources

protected void doCleanUpResources()

Overrides:

doCleanUpResources in class AbstractFSWAL<WALProvider.Writer>

shutDownSyncRunners

private void shutDownSyncRunners()

doWriterSync

protected CompletableFuture<Long> doWriterSync(WALProvider.Writer writer,
 boolean shouldUseHSync,
 long txidWhenSync)

Specified by:

doWriterSync in class AbstractFSWAL<WALProvider.Writer>

offerSyncRequest

private void offerSyncRequest(FSHLog.SyncRequest syncRequest)

checkSlowSyncCount

protected void checkSlowSyncCount()

Overrides:

checkSlowSyncCount in class AbstractFSWAL<WALProvider.Writer>

doCheckLogLowReplication

protected boolean doCheckLogLowReplication()

Returns true if number of replicas for the WAL is lower than threshold

Specified by:

doCheckLogLowReplication in class AbstractFSWAL<WALProvider.Writer>

getLogReplication

int getLogReplication()

This method gets the datanode replication count for the current WAL.

If the pipeline isn't started yet or is empty, you will get the default replication factor. Therefore, if this function returns 0, it means you are not properly running with the HDFS-826 patch.

Specified by:

getLogReplication in class AbstractFSWAL<WALProvider.Writer>

isLowReplicationRollEnabled

boolean isLowReplicationRollEnabled()

getPipeline

org.apache.hadoop.hdfs.protocol.DatanodeInfo[] getPipeline()

This method gets the pipeline for the current WAL.

Specified by:

getPipeline in class AbstractFSWAL<WALProvider.Writer>

createCombinedWriter

protected WALProvider.Writer createCombinedWriter(WALProvider.Writer localWriter,
 WALProvider.Writer remoteWriter)

Specified by:

createCombinedWriter in class AbstractFSWAL<WALProvider.Writer>