org.apache.hadoop.hbase.regionserver.wal

Class FSHLog

java.lang.Object

org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL<WALProvider.Writer>

org.apache.hadoop.hbase.regionserver.wal.FSHLog

All Implemented Interfaces:

Closeable, AutoCloseable, WALFileLengthProvider, WAL

@InterfaceAudience.Private
public class FSHLog
extends AbstractFSWAL<WALProvider.Writer>

The original implementation of FSWAL.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
(package private) static class	FSHLog.SyncRequest
private class	FSHLog.SyncRunner Thread to runs the hdfs sync call.

Nested classes/interfaces inherited from interface org.apache.hadoop.hbase.wal.WAL

WAL.Entry

Field Summary

Fields

Modifier and Type	Field and Description
private AtomicInteger	consecutiveLogRolls
private static int	DEFAULT_FSHLOG_WAIT_ON_SHUTDOWN_IN_SECONDS
private static int	DEFAULT_LOW_REPLICATION_ROLL_LIMIT
private static int	DEFAULT_MAX_BATCH_COUNT
private static int	DEFAULT_SYNCER_COUNT
static long	FIXED_OVERHEAD
private static String	FSHLOG_WAIT_ON_SHUTDOWN_IN_SECONDS
private org.apache.hadoop.fs.FSDataOutputStream	hdfs_out FSDataOutputStream associated with the current SequenceFile.writer
private static org.slf4j.Logger	LOG
private static String	LOW_REPLICATION_ROLL_LIMIT
private boolean	lowReplicationRollEnabled
private int	lowReplicationRollLimit
private static String	MAX_BATCH_COUNT
private int	maxSyncRequestCount
private int	minTolerableReplication
private static String	SYNCER_COUNT
private int	syncerCount
private int	syncRunnerIndex Which syncrunner to use next.
private FSHLog.SyncRunner[]	syncRunners
private static String	TOLERABLE_LOW_REPLICATION
private static IOException	WAL_CLOSE_EXCEPTION
private static IOException	WITER_BROKEN_EXCEPTION
private static IOException	WITER_REPLACED_EXCEPTION

Fields inherited from class org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL

abortable, blocksize, closed, closeExecutor, conf, consumeExecutor, consumer, coprocessorHost, DEFAULT_ROLL_ON_SYNC_TIME_MS, DEFAULT_SLOW_SYNC_ROLL_INTERVAL_MS, DEFAULT_SLOW_SYNC_ROLL_THRESHOLD, DEFAULT_SLOW_SYNC_TIME_MS, DEFAULT_WAL_BATCH_SIZE, DEFAULT_WAL_SHUTDOWN_WAIT_TIMEOUT_MS, DEFAULT_WAL_SYNC_TIMEOUT_MS, filenum, fs, hasConsumerTask, highestProcessedAppendTxid, highestSyncedTxid, highestUnsyncedTxid, implClassName, inflightWALClosures, listeners, LOG_NAME_COMPARATOR, logrollsize, MAX_LOGS, maxLogs, numEntries, ourFiles, prefixPathStr, RING_BUFFER_SLOT_COUNT, ROLL_ON_SYNC_TIME_MS, rollOnSyncNs, rollRequested, rollWriterLock, sequenceIdAccounting, shouldShutDownConsumeExecutorWhenClose, shutdown, SLOW_SYNC_ROLL_INTERVAL_MS, SLOW_SYNC_ROLL_THRESHOLD, SLOW_SYNC_TIME_MS, slowSyncCheckInterval, slowSyncCount, slowSyncNs, slowSyncRollThreshold, syncFutureCache, syncFutures, totalLogSize, toWriteAppends, unackedAppends, useHsync, WAL_BATCH_SIZE, WAL_ROLL_MULTIPLIER, WAL_SHUTDOWN_WAIT_TIMEOUT_MS, WAL_SYNC_TIMEOUT_MS, walArchiveDir, walDir, walFile2Props, walFilePrefix, walFileSuffix, walShutdownTimeout, writer

Constructor Summary

Constructors

Constructor and Description
FSHLog(org.apache.hadoop.fs.FileSystem fs, Abortable abortable, org.apache.hadoop.fs.Path root, String logDir, org.apache.hadoop.conf.Configuration conf)
FSHLog(org.apache.hadoop.fs.FileSystem fs, Abortable abortable, org.apache.hadoop.fs.Path rootDir, String logDir, String archiveDir, org.apache.hadoop.conf.Configuration conf, List<WALActionsListener> listeners, boolean failIfWALExists, String prefix, String suffix, org.apache.hadoop.fs.FileSystem remoteFs, org.apache.hadoop.fs.Path remoteWALDir) Create an edit log at the given dir location.
FSHLog(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path root, String logDir, org.apache.hadoop.conf.Configuration conf) Constructor.
FSHLog(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path rootDir, String logDir, String archiveDir, org.apache.hadoop.conf.Configuration conf, List<WALActionsListener> listeners, boolean failIfWALExists, String prefix, String suffix)

Method Summary

Modifier and Type	Method and Description
protected void	checkSlowSyncCount()
protected WALProvider.Writer	createCombinedWriter(WALProvider.Writer localWriter, WALProvider.Writer remoteWriter)
private void	createSyncRunnersAndStart()
protected WALProvider.Writer	createWriterInstance(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path) This method allows subclasses to inject different writers without having to extend other methods like rollWriter().
protected void	doAppend(WALProvider.Writer writer, FSWALEntry entry)
protected boolean	doCheckLogLowReplication() Returns true if number of replicas for the WAL is lower than threshold
protected void	doCleanUpResources()
protected CompletableFuture<Long>	doWriterSync(WALProvider.Writer writer, boolean shouldUseHSync, long txidWhenSync)
(package private) int	getLogReplication() This method gets the datanode replication count for the current WAL.
(package private) OutputStream	getOutputStream() Currently, we need to expose the writer's OutputStream to tests so that they can manipulate the default behavior (such as setting the maxRecoveryErrorCount value).
(package private) org.apache.hadoop.hdfs.protocol.DatanodeInfo[]	getPipeline() This method gets the pipeline for the current WAL.
(package private) WALProvider.Writer	getWriter()
void	init() Used to initialize the WAL.
(package private) boolean	isLowReplicationRollEnabled()
private void	offerSyncRequest(FSHLog.SyncRequest syncRequest)
protected void	onWriterReplaced(WALProvider.Writer nextWriter)
private void	preemptiveSync(ProtobufLogWriter nextWriter) Run a sync after opening to set up the pipeline.
(package private) void	setWriter(WALProvider.Writer writer)
private void	shutDownSyncRunners()

Methods inherited from class org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL

abortCacheFlush, append, appendData, appendEntry, appendMarker, archive, archiveLogFile, atHeadOfRingBufferEventHandlerAppend, blockOnSync, checkLogLowReplication, close, closeWriter, completeCacheFlush, computeFilename, createSingleThreadPoolConsumeExecutor, doCheckSlowSync, doReplaceWriter, doShutdown, doSync, doSync, findRegionsToForceFlush, getCoprocessorHost, getCurrentFileName, getEarliestMemStoreSeqNum, getEarliestMemStoreSeqNum, getFilenum, getFileNumFromFileName, getFiles, getInflightWALCloseCount, getLogFileSize, getLogFileSizeIfBeingWritten, getNumLogFiles, getNumRolledLogFiles, getOldPath, getPreallocatedEventCount, getSyncedTxid, getSyncFuture, getUnflushedEntriesCount, getWALArchivePath, isLogRollRequested, isUnflushedEntries, isWriterBroken, logRollAndSetupWalProps, main, markFutureDoneAndOffer, postSync, registerWALActionsListener, replaceWriter, requestLogRoll, requestLogRoll, rollWriter, rollWriter, setWaitOnShutdownInSeconds, shutdown, skipRemoteWAL, stampSequenceIdAndPublishToRingBuffer, startCacheFlush, startCacheFlush, sync, sync, sync, sync, toString, unregisterWALActionsListener, updateStore, waitForSafePoint

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

LOG

private static final org.slf4j.Logger LOG

TOLERABLE_LOW_REPLICATION

private static final String TOLERABLE_LOW_REPLICATION

See Also:

Constant Field Values

LOW_REPLICATION_ROLL_LIMIT

private static final String LOW_REPLICATION_ROLL_LIMIT

See Also:

Constant Field Values

DEFAULT_LOW_REPLICATION_ROLL_LIMIT

private static final int DEFAULT_LOW_REPLICATION_ROLL_LIMIT

See Also:

Constant Field Values

SYNCER_COUNT

private static final String SYNCER_COUNT

See Also:

Constant Field Values

DEFAULT_SYNCER_COUNT

private static final int DEFAULT_SYNCER_COUNT

See Also:

Constant Field Values

MAX_BATCH_COUNT

private static final String MAX_BATCH_COUNT

See Also:

Constant Field Values

DEFAULT_MAX_BATCH_COUNT

private static final int DEFAULT_MAX_BATCH_COUNT

See Also:

Constant Field Values

FSHLOG_WAIT_ON_SHUTDOWN_IN_SECONDS

private static final String FSHLOG_WAIT_ON_SHUTDOWN_IN_SECONDS

See Also:

Constant Field Values

DEFAULT_FSHLOG_WAIT_ON_SHUTDOWN_IN_SECONDS

private static final int DEFAULT_FSHLOG_WAIT_ON_SHUTDOWN_IN_SECONDS

See Also:

Constant Field Values

WITER_REPLACED_EXCEPTION

private static final IOException WITER_REPLACED_EXCEPTION

WITER_BROKEN_EXCEPTION

private static final IOException WITER_BROKEN_EXCEPTION

WAL_CLOSE_EXCEPTION

private static final IOException WAL_CLOSE_EXCEPTION

hdfs_out

private org.apache.hadoop.fs.FSDataOutputStream hdfs_out

FSDataOutputStream associated with the current SequenceFile.writer

minTolerableReplication

private final int minTolerableReplication

consecutiveLogRolls

private final AtomicInteger consecutiveLogRolls

lowReplicationRollLimit

private final int lowReplicationRollLimit

lowReplicationRollEnabled

private volatile boolean lowReplicationRollEnabled

syncerCount

private final int syncerCount

maxSyncRequestCount

private final int maxSyncRequestCount

syncRunnerIndex

private int syncRunnerIndex

Which syncrunner to use next.

syncRunners

private FSHLog.SyncRunner[] syncRunners

FIXED_OVERHEAD

public static final long FIXED_OVERHEAD

Constructor Detail

FSHLog

public FSHLog(org.apache.hadoop.fs.FileSystem fs,
              org.apache.hadoop.fs.Path root,
              String logDir,
              org.apache.hadoop.conf.Configuration conf)
       throws IOException

Constructor.

Parameters:

fs - filesystem handle

root - path for stored and archived wals

logDir - dir where wals are stored

conf - configuration to use

Throws:

IOException

FSHLog

public FSHLog(org.apache.hadoop.fs.FileSystem fs,
              Abortable abortable,
              org.apache.hadoop.fs.Path root,
              String logDir,
              org.apache.hadoop.conf.Configuration conf)
       throws IOException

Throws:

IOException

FSHLog

public FSHLog(org.apache.hadoop.fs.FileSystem fs,
              org.apache.hadoop.fs.Path rootDir,
              String logDir,
              String archiveDir,
              org.apache.hadoop.conf.Configuration conf,
              List<WALActionsListener> listeners,
              boolean failIfWALExists,
              String prefix,
              String suffix)
       throws IOException

Throws:

IOException

FSHLog

public FSHLog(org.apache.hadoop.fs.FileSystem fs,
              Abortable abortable,
              org.apache.hadoop.fs.Path rootDir,
              String logDir,
              String archiveDir,
              org.apache.hadoop.conf.Configuration conf,
              List<WALActionsListener> listeners,
              boolean failIfWALExists,
              String prefix,
              String suffix,
              org.apache.hadoop.fs.FileSystem remoteFs,
              org.apache.hadoop.fs.Path remoteWALDir)
       throws IOException

Create an edit log at the given dir location. You should never have to load an existing log. If there is a log at startup, it should have already been processed and deleted by the time the WAL object is started up.

Parameters:

fs - filesystem handle

abortable - Abortable - the server here

rootDir - path to where logs and oldlogs

logDir - dir where wals are stored

archiveDir - dir where wals are archived

conf - configuration to use

listeners - Listeners on WAL events. Listeners passed here will be registered before we do anything else; e.g. the Constructor AbstractFSWAL.rollWriter().

failIfWALExists - If true IOException will be thrown if files related to this wal already exist.

prefix - should always be hostname and port in distributed env and it will be URL encoded before being used. If prefix is null, "wal" will be used

suffix - will be url encoded. null is treated as empty. non-empty must start with AbstractFSWALProvider.WAL_FILE_NAME_DELIMITER

Throws:

IOException

Method Detail

init

public void init()
          throws IOException

Description copied from class: AbstractFSWAL

Used to initialize the WAL. Usually just call rollWriter to create the first log writer.

Specified by:

init in interface WAL

Overrides:

init in class AbstractFSWAL<WALProvider.Writer>

Throws:

IOException

createSyncRunnersAndStart

private void createSyncRunnersAndStart()

getOutputStream

OutputStream getOutputStream()

Currently, we need to expose the writer's OutputStream to tests so that they can manipulate the default behavior (such as setting the maxRecoveryErrorCount value). This is done using reflection on the underlying HDFS OutputStream. NOTE: This could be removed once Hadoop1 support is removed.

Returns:

null if underlying stream is not ready.

preemptiveSync

private void preemptiveSync(ProtobufLogWriter nextWriter)

Run a sync after opening to set up the pipeline.

createWriterInstance

protected WALProvider.Writer createWriterInstance(org.apache.hadoop.fs.FileSystem fs,
                                                  org.apache.hadoop.fs.Path path)
                                           throws IOException

This method allows subclasses to inject different writers without having to extend other methods like rollWriter().

Specified by:

createWriterInstance in class AbstractFSWAL<WALProvider.Writer>

Returns:

Writer instance

Throws:

IOException

doAppend

protected void doAppend(WALProvider.Writer writer,
                        FSWALEntry entry)
                 throws IOException

Specified by:

doAppend in class AbstractFSWAL<WALProvider.Writer>

Throws:

IOException

onWriterReplaced

protected void onWriterReplaced(WALProvider.Writer nextWriter)

Specified by:

onWriterReplaced in class AbstractFSWAL<WALProvider.Writer>

doCleanUpResources

protected void doCleanUpResources()

Overrides:

doCleanUpResources in class AbstractFSWAL<WALProvider.Writer>

shutDownSyncRunners

private void shutDownSyncRunners()

doWriterSync

protected CompletableFuture<Long> doWriterSync(WALProvider.Writer writer,
                                               boolean shouldUseHSync,
                                               long txidWhenSync)

Specified by:

doWriterSync in class AbstractFSWAL<WALProvider.Writer>

offerSyncRequest

private void offerSyncRequest(FSHLog.SyncRequest syncRequest)

checkSlowSyncCount

protected void checkSlowSyncCount()

Overrides:

checkSlowSyncCount in class AbstractFSWAL<WALProvider.Writer>

doCheckLogLowReplication

protected boolean doCheckLogLowReplication()

Returns true if number of replicas for the WAL is lower than threshold

Specified by:

doCheckLogLowReplication in class AbstractFSWAL<WALProvider.Writer>

getLogReplication

int getLogReplication()

This method gets the datanode replication count for the current WAL.

If the pipeline isn't started yet or is empty, you will get the default replication factor. Therefore, if this function returns 0, it means you are not properly running with the HDFS-826 patch.

Specified by:

getLogReplication in class AbstractFSWAL<WALProvider.Writer>

isLowReplicationRollEnabled

boolean isLowReplicationRollEnabled()

getPipeline

org.apache.hadoop.hdfs.protocol.DatanodeInfo[] getPipeline()

This method gets the pipeline for the current WAL.

Specified by:

getPipeline in class AbstractFSWAL<WALProvider.Writer>

getWriter

WALProvider.Writer getWriter()

setWriter

void setWriter(WALProvider.Writer writer)

createCombinedWriter

protected WALProvider.Writer createCombinedWriter(WALProvider.Writer localWriter,
                                                  WALProvider.Writer remoteWriter)

Specified by:

createCombinedWriter in class AbstractFSWAL<WALProvider.Writer>