WALProcedureStore (Apache HBase 2.2.3 API)

java.lang.Object
- org.apache.hadoop.hbase.procedure2.store.ProcedureStoreBase
- - org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore

All Implemented Interfaces:

ProcedureStore
```
@InterfaceAudience.Private
public class WALProcedureStore
extends ProcedureStoreBase
```
WAL implementation of the ProcedureStore.
When starting, the upper layer will first call start(int), then recoverLease(), then ProcedureStore.load(ProcedureLoader).
In recoverLease(), we will get the lease by closing all the existing wal files(by calling recoverFileLease), and creating a new wal writer. And we will also get the list of all the old wal files.
FIXME: notice that the current recover lease implementation is problematic, it can not deal with the races if there are two master both wants to acquire the lease...
In ProcedureStore.load(ProcedureLoader) method, we will load all the active procedures. See the comments of this method for more details.
The actual logging way is a bit like our FileSystem based WAL implementation as RS side. There is a slots, which is more like the ring buffer, and in the insert, update and delete methods we will put thing into the slots and wait. And there is a background sync thread(see the syncLoop() method) which get data from the slots and write them to the FileSystem, and notify the caller that we have finished.
TODO: try using disruptor to increase performance and simplify the logic?
The storeTracker keeps track of the modified procedures in the newest wal file, which is also the one being written currently. And the deleted bits in it are for all the procedures, not only the ones in the newest wal file. And when rolling a log, we will first store it in the trailer of the current wal file, and then reset its modified bits, so that it can start to track the modified procedures for the new wal file.
The holdingCleanupTracker is used to test whether we are safe to delete the oldest wal file. When there are log rolling and there are more than 1 wal files, we will make use of it. It will first be initialized to the oldest file's tracker(which is stored in the trailer), using the method ProcedureStoreTracker.resetTo(ProcedureStoreTracker, boolean), and then merge it with the tracker of every newer wal files, using the ProcedureStoreTracker.setDeletedIfModifiedInBoth(ProcedureStoreTracker). If we find out that all the modified procedures for the oldest wal file are modified or deleted in newer wal files, then we can delete it. This is because that, every time we call ProcedureStore.insert(Procedure[]) or ProcedureStore.update(Procedure), we will persist the full state of a Procedure, so the earlier wal records for this procedure can all be deleted.

See Also:

for printing content of a single WAL., to parse a directory of MasterWALProcs.

Nested Class Summary

Nested Classes
Modifier and Type	Class and Description
`static interface`	`WALProcedureStore.LeaseRecovery`
`private static class`	`WALProcedureStore.PushType`
`static class`	`WALProcedureStore.SyncMetrics`

Nested classes/interfaces inherited from interface org.apache.hadoop.hbase.procedure2.store.ProcedureStore
ProcedureStore.ProcedureIterator, ProcedureStore.ProcedureLoader, ProcedureStore.ProcedureStoreListener

Field Summary

Fields
Modifier and Type	Field and Description
`private org.apache.hadoop.conf.Configuration`	`conf`
`private Set<ProcedureWALFile>`	`corruptedLogs`
`private static boolean`	`DEFAULT_EXEC_WAL_CLEANUP_ON_LOAD_CONF_KEY`
`private static int`	`DEFAULT_MAX_RETRIES_BEFORE_ROLL`
`private static int`	`DEFAULT_MAX_SYNC_FAILURE_ROLL`
`private static int`	`DEFAULT_PERIODIC_ROLL`
`private static int`	`DEFAULT_ROLL_RETRIES`
`private static long`	`DEFAULT_ROLL_THRESHOLD`
`private static int`	`DEFAULT_SYNC_STATS_COUNT`
`private static int`	`DEFAULT_SYNC_WAIT_MSEC`
`private static boolean`	`DEFAULT_USE_HSYNC`
`private static int`	`DEFAULT_WAIT_BEFORE_ROLL`
`private static int`	`DEFAULT_WAL_COUNT_WARN_THRESHOLD`
`private boolean`	`enforceStreamCapability`
`static String`	`EXEC_WAL_CLEANUP_ON_LOAD_CONF_KEY`
`private static Comparator<org.apache.hadoop.fs.FileStatus>`	`FILE_STATUS_ID_COMPARATOR`
`private long`	`flushLogId`
`private org.apache.hadoop.fs.FileSystem`	`fs`
`private ProcedureStoreTracker`	`holdingCleanupTracker`
`private AtomicBoolean`	`inSync`
`private AtomicLong`	`lastRollTs`
`private WALProcedureStore.LeaseRecovery`	`leaseRecovery`
`private AtomicBoolean`	`loading`
`private ReentrantLock`	`lock`
`private static org.slf4j.Logger`	`LOG`
`static String`	`LOG_PREFIX`
`private LinkedList<ProcedureWALFile>`	`logs`
`static String`	`MASTER_PROCEDURE_LOGDIR` Used to construct the name of the log directory for master procedures
`static String`	`MAX_RETRIES_BEFORE_ROLL_CONF_KEY`
`static String`	`MAX_SYNC_FAILURE_ROLL_CONF_KEY`
`private int`	`maxRetriesBeforeRoll`
`private int`	`maxSyncFailureRoll`
`static String`	`PERIODIC_ROLL_CONF_KEY`
`private int`	`periodicRollMsec`
`static String`	`ROLL_RETRIES_CONF_KEY`
`static String`	`ROLL_THRESHOLD_CONF_KEY`
`private int`	`rollRetries`
`private long`	`rollThreshold`
`private int`	`runningProcCount`
`private Condition`	`slotCond`
`private int`	`slotIndex`
`private ByteSlot[]`	`slots`
`private LinkedTransferQueue<ByteSlot>`	`slotsCache`
`static String`	`STORE_WAL_SYNC_STATS_COUNT`
`private ProcedureStoreTracker`	`storeTracker`
`private org.apache.hadoop.fs.FSDataOutputStream`	`stream`
`static String`	`SYNC_WAIT_MSEC_CONF_KEY`
`private Condition`	`syncCond`
`private AtomicReference<Throwable>`	`syncException`
`private AtomicLong`	`syncId`
`private int`	`syncMaxSlot`
`private org.apache.hbase.thirdparty.org.apache.commons.collections4.queue.CircularFifoQueue<WALProcedureStore.SyncMetrics>`	`syncMetricsQueue`
`private Thread`	`syncThread`
`private int`	`syncWaitMsec`
`private AtomicLong`	`totalSynced`
`static String`	`USE_HSYNC_CONF_KEY`
`private boolean`	`useHsync`
`static String`	`WAIT_BEFORE_ROLL_CONF_KEY`
`private int`	`waitBeforeRoll`
`private Condition`	`waitCond`
`static String`	`WAL_COUNT_WARN_THRESHOLD_CONF_KEY`
`private org.apache.hadoop.fs.Path`	`walArchiveDir`
`private int`	`walCountWarnThreshold`
`private org.apache.hadoop.fs.Path`	`walDir`
`private static org.apache.hadoop.fs.PathFilter`	`WALS_PATH_FILTER`

Constructor Summary

Constructors
Constructor and Description
`WALProcedureStore(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path walDir, org.apache.hadoop.fs.Path walArchiveDir, WALProcedureStore.LeaseRecovery leaseRecovery)`
`WALProcedureStore(org.apache.hadoop.conf.Configuration conf, WALProcedureStore.LeaseRecovery leaseRecovery)`

Method Summary

All Methods Static Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`private ByteSlot`	`acquireSlot()`
`private void`	`buildHoldingCleanupTracker()`
`private void`	`closeCurrentLogStream(boolean abort)`
`void`	`delete(long procId)` The specified procId was removed from the executor, due to completion, abort or failure.
`private void`	`delete(long[] procIds)`
`void`	`delete(long[] procIds, int offset, int count)` The specified procIds were removed from the executor, due to completion, abort or failure.
`void`	`delete(Procedure<?> proc, long[] subProcIds)` The parent procedure completed.
`ArrayList<ProcedureWALFile>`	`getActiveLogs()`
`Set<ProcedureWALFile>`	`getCorruptedLogs()`
`org.apache.hadoop.fs.FileSystem`	`getFileSystem()`
`protected org.apache.hadoop.fs.Path`	`getLogFilePath(long logId)`
`private org.apache.hadoop.fs.FileStatus[]`	`getLogFiles()`
`private static long`	`getLogIdFromName(String name)`
`private static long`	`getMaxLogId(org.apache.hadoop.fs.FileStatus[] logFiles)` Make sure that the file set are gotten by calling `getLogFiles()`, where we will sort the file set by log id.
`long`	`getMillisFromLastRoll()`
`long`	`getMillisToNextPeriodicRoll()`
`int`	`getNumThreads()`
`ProcedureStoreTracker`	`getStoreTracker()`
`ArrayList<WALProcedureStore.SyncMetrics>`	`getSyncMetrics()`
`(package private) org.apache.hadoop.fs.Path`	`getWalArchiveDir()`
`org.apache.hadoop.fs.Path`	`getWALDir()`
`private ProcedureWALFile`	`initOldLog(org.apache.hadoop.fs.FileStatus logFile, org.apache.hadoop.fs.Path walArchiveDir)` Loads given log file and it's tracker.
`private long`	`initOldLogs(org.apache.hadoop.fs.FileStatus[] logFiles)` Make sure that the file set are gotten by calling `getLogFiles()`, where we will sort the file set by log id.
`private void`	`initTrackerFromOldLogs()` If last log's tracker is not null, use it as `storeTracker`.
`void`	`insert(Procedure<?>[] procs)` Serialize a set of new procedures.
`void`	`insert(Procedure<?> proc, Procedure<?>[] subprocs)` When a procedure is submitted to the executor insert(proc, null) will be called.
`private boolean`	`isSyncAborted()`
`void`	`load(ProcedureStore.ProcedureLoader loader)` Load the Procedures in the store.
`static void`	`main(String[] args)` Parses a directory of WALs building up ProcedureState.
`private void`	`periodicRoll()`
`(package private) void`	`periodicRollForTesting()`
`private long`	`pushData(WALProcedureStore.PushType type, ByteSlot slot, long procId, long[] subProcIds)`
`void`	`recoverLease()` Acquire the lease for the procedure store.
`private void`	`releaseSlot(ByteSlot slot)`
`private void`	`removeAllLogs(long lastLogId, String why)` Remove all logs with logId <= `lastLogId`.
`private void`	`removeInactiveLogs()`
`(package private) void`	`removeInactiveLogsForTesting()`
`private boolean`	`removeLogFile(ProcedureWALFile log, org.apache.hadoop.fs.Path walArchiveDir)`
`private boolean`	`rollWriter()`
`(package private) boolean`	`rollWriter(long logId)`
`boolean`	`rollWriterForTesting()`
`private boolean`	`rollWriterWithRetries()`
`private void`	`sendStopSignal()`
`int`	`setRunningProcedureCount(int count)` Set the number of procedure running.
`void`	`start(int numSlots)` Start/Open the procedure store
`void`	`stop(boolean abort)` Stop/Close the procedure store
`private void`	`syncLoop()`
`private long`	`syncSlots()`
`protected long`	`syncSlots(org.apache.hadoop.fs.FSDataOutputStream stream, ByteSlot[] slots, int offset, int count)`
`protected void`	`syncStream(org.apache.hadoop.fs.FSDataOutputStream stream)`
`private void`	`tryCleanupLogsOnLoad()`
`private boolean`	`tryRollWriter()`
`void`	`update(Procedure<?> proc)` The specified procedure was executed, and the new state should be written to the store.
`private void`	`updateStoreTracker(WALProcedureStore.PushType type, long procId, long[] subProcIds)`

Methods inherited from class org.apache.hadoop.hbase.procedure2.store.ProcedureStoreBase
isRunning, registerListener, sendAbortProcessSignal, sendForceUpdateSignal, sendPostSyncSignal, setRunning, unregisterListener

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

private static final org.slf4j.Logger LOG

LOG_PREFIX
```
public static final String LOG_PREFIX
```
See Also:

Constant Field Values

MASTER_PROCEDURE_LOGDIR
```
public static final String MASTER_PROCEDURE_LOGDIR
```
Used to construct the name of the log directory for master procedures

See Also:

Constant Field Values

WAL_COUNT_WARN_THRESHOLD_CONF_KEY

public static final String WAL_COUNT_WARN_THRESHOLD_CONF_KEY

See Also:: Constant Field Values

DEFAULT_WAL_COUNT_WARN_THRESHOLD
```
private static final int DEFAULT_WAL_COUNT_WARN_THRESHOLD
```
See Also:

Constant Field Values

EXEC_WAL_CLEANUP_ON_LOAD_CONF_KEY

public static final String EXEC_WAL_CLEANUP_ON_LOAD_CONF_KEY

See Also:: Constant Field Values

DEFAULT_EXEC_WAL_CLEANUP_ON_LOAD_CONF_KEY

private static final boolean DEFAULT_EXEC_WAL_CLEANUP_ON_LOAD_CONF_KEY

See Also:: Constant Field Values

MAX_RETRIES_BEFORE_ROLL_CONF_KEY

public static final String MAX_RETRIES_BEFORE_ROLL_CONF_KEY

See Also:: Constant Field Values

DEFAULT_MAX_RETRIES_BEFORE_ROLL
```
private static final int DEFAULT_MAX_RETRIES_BEFORE_ROLL
```
See Also:

Constant Field Values

WAIT_BEFORE_ROLL_CONF_KEY

public static final String WAIT_BEFORE_ROLL_CONF_KEY

See Also:: Constant Field Values

DEFAULT_WAIT_BEFORE_ROLL
```
private static final int DEFAULT_WAIT_BEFORE_ROLL
```
See Also:

Constant Field Values

ROLL_RETRIES_CONF_KEY

public static final String ROLL_RETRIES_CONF_KEY

See Also:: Constant Field Values

DEFAULT_ROLL_RETRIES
```
private static final int DEFAULT_ROLL_RETRIES
```
See Also:

Constant Field Values

MAX_SYNC_FAILURE_ROLL_CONF_KEY

public static final String MAX_SYNC_FAILURE_ROLL_CONF_KEY

See Also:: Constant Field Values

DEFAULT_MAX_SYNC_FAILURE_ROLL
```
private static final int DEFAULT_MAX_SYNC_FAILURE_ROLL
```
See Also:

Constant Field Values

PERIODIC_ROLL_CONF_KEY

public static final String PERIODIC_ROLL_CONF_KEY

See Also:: Constant Field Values

DEFAULT_PERIODIC_ROLL
```
private static final int DEFAULT_PERIODIC_ROLL
```
See Also:

Constant Field Values

SYNC_WAIT_MSEC_CONF_KEY

public static final String SYNC_WAIT_MSEC_CONF_KEY

See Also:: Constant Field Values

DEFAULT_SYNC_WAIT_MSEC
```
private static final int DEFAULT_SYNC_WAIT_MSEC
```
See Also:

Constant Field Values

USE_HSYNC_CONF_KEY

public static final String USE_HSYNC_CONF_KEY

See Also:: Constant Field Values

DEFAULT_USE_HSYNC

private static final boolean DEFAULT_USE_HSYNC

See Also:: Constant Field Values

ROLL_THRESHOLD_CONF_KEY

public static final String ROLL_THRESHOLD_CONF_KEY

See Also:: Constant Field Values

DEFAULT_ROLL_THRESHOLD
```
private static final long DEFAULT_ROLL_THRESHOLD
```
See Also:

Constant Field Values

STORE_WAL_SYNC_STATS_COUNT

public static final String STORE_WAL_SYNC_STATS_COUNT

See Also:: Constant Field Values

DEFAULT_SYNC_STATS_COUNT
```
private static final int DEFAULT_SYNC_STATS_COUNT
```
See Also:

Constant Field Values

logs

private final LinkedList<ProcedureWALFile> logs

holdingCleanupTracker

private final ProcedureStoreTracker holdingCleanupTracker

storeTracker

private final ProcedureStoreTracker storeTracker

lock
```
private final ReentrantLock lock
```

waitCond
```
private final Condition waitCond
```

slotCond
```
private final Condition slotCond
```

syncCond
```
private final Condition syncCond
```

leaseRecovery

private final WALProcedureStore.LeaseRecovery leaseRecovery

conf

private final org.apache.hadoop.conf.Configuration conf

fs

private final org.apache.hadoop.fs.FileSystem fs

walDir

private final org.apache.hadoop.fs.Path walDir

walArchiveDir

private final org.apache.hadoop.fs.Path walArchiveDir

enforceStreamCapability

private final boolean enforceStreamCapability

syncException

private final AtomicReference<Throwable> syncException

loading
```
private final AtomicBoolean loading
```

inSync
```
private final AtomicBoolean inSync
```

totalSynced
```
private final AtomicLong totalSynced
```

lastRollTs
```
private final AtomicLong lastRollTs
```

syncId
```
private final AtomicLong syncId
```

slotsCache

private LinkedTransferQueue<ByteSlot> slotsCache

corruptedLogs

private Set<ProcedureWALFile> corruptedLogs

stream

private org.apache.hadoop.fs.FSDataOutputStream stream

runningProcCount
```
private int runningProcCount
```

flushLogId
```
private long flushLogId
```

syncMaxSlot
```
private int syncMaxSlot
```

slotIndex
```
private int slotIndex
```

syncThread
```
private Thread syncThread
```

slots
```
private ByteSlot[] slots
```

walCountWarnThreshold
```
private int walCountWarnThreshold
```

maxRetriesBeforeRoll
```
private int maxRetriesBeforeRoll
```

maxSyncFailureRoll
```
private int maxSyncFailureRoll
```

waitBeforeRoll
```
private int waitBeforeRoll
```

rollRetries
```
private int rollRetries
```

periodicRollMsec
```
private int periodicRollMsec
```

rollThreshold
```
private long rollThreshold
```

useHsync
```
private boolean useHsync
```

syncWaitMsec
```
private int syncWaitMsec
```

syncMetricsQueue

private org.apache.hbase.thirdparty.org.apache.commons.collections4.queue.CircularFifoQueue<WALProcedureStore.SyncMetrics> syncMetricsQueue

WALS_PATH_FILTER

private static final org.apache.hadoop.fs.PathFilter WALS_PATH_FILTER

FILE_STATUS_ID_COMPARATOR

private static final Comparator<org.apache.hadoop.fs.FileStatus> FILE_STATUS_ID_COMPARATOR

Constructor Detail

WALProcedureStore

public WALProcedureStore(org.apache.hadoop.conf.Configuration conf,
                         WALProcedureStore.LeaseRecovery leaseRecovery)
                  throws IOException

Throws:: IOException

WALProcedureStore

public WALProcedureStore(org.apache.hadoop.conf.Configuration conf,
                         org.apache.hadoop.fs.Path walDir,
                         org.apache.hadoop.fs.Path walArchiveDir,
                         WALProcedureStore.LeaseRecovery leaseRecovery)
                  throws IOException

Throws:: IOException

Method Detail
- start
```
public void start(int numSlots)
           throws IOException
```
  Description copied from interface: ProcedureStore
  
  Start/Open the procedure store
  
  Parameters:
  
  numSlots - number of threads to be used by the procedure store
  
  Throws:
  
  IOException
- stop
```
public void stop(boolean abort)
```
  Description copied from interface: ProcedureStore
  
  Stop/Close the procedure store
  
  Parameters:
  
  abort - true if the stop is an abort
- sendStopSignal
```
private void sendStopSignal()
```
- getNumThreads
```
public int getNumThreads()
```
  Returns:
  
  the number of threads/slots passed to start()
- setRunningProcedureCount
```
public int setRunningProcedureCount(int count)
```
  Description copied from interface: ProcedureStore
  
  Set the number of procedure running. This can be used, for example, by the store to know how long to wait before a sync.
  
  Returns:
  
  how many procedures are running (may not be same as count).
- getStoreTracker
```
public ProcedureStoreTracker getStoreTracker()
```
- getActiveLogs
```
public ArrayList<ProcedureWALFile> getActiveLogs()
```
- getCorruptedLogs
```
public Set<ProcedureWALFile> getCorruptedLogs()
```
- recoverLease
```
public void recoverLease()
                  throws IOException
```
  Description copied from interface: ProcedureStore
  
  Acquire the lease for the procedure store.
  
  Throws:
  
  IOException
- load
```
public void load(ProcedureStore.ProcedureLoader loader)
          throws IOException
```
  Description copied from interface: ProcedureStore
  
  Load the Procedures in the store.
  
  Parameters:
  
  loader - the ProcedureLoader that will handle the store-load events
  
  Throws:
  
  IOException
- tryCleanupLogsOnLoad
```
private void tryCleanupLogsOnLoad()
```
- insert
```
public void insert(Procedure<?> proc,
                   Procedure<?>[] subprocs)
```
  Description copied from interface: ProcedureStore
  
  When a procedure is submitted to the executor insert(proc, null) will be called. 'proc' has a 'RUNNABLE' state and the initial information required to start up. When a procedure is executed and it returns children insert(proc, subprocs) will be called. 'proc' has a 'WAITING' state and an update state. 'subprocs' are the children in 'RUNNABLE' state with the initial information.
  
  Parameters:
  
  proc - the procedure to serialize and write to the store.
  
  subprocs - the newly created child of the proc.
- insert
```
public void insert(Procedure<?>[] procs)
```
  Description copied from interface: ProcedureStore
  
  Serialize a set of new procedures. These procedures are freshly submitted to the executor and each procedure has a 'RUNNABLE' state and the initial information required to start up.
  
  Parameters:
  
  procs - the procedures to serialize and write to the store.
- update
```
public void update(Procedure<?> proc)
```
  Description copied from interface: ProcedureStore
  
  The specified procedure was executed, and the new state should be written to the store.
  
  Parameters:
  
  proc - the procedure to serialize and write to the store.
- delete
```
public void delete(long procId)
```
  Description copied from interface: ProcedureStore
  
  The specified procId was removed from the executor, due to completion, abort or failure. The store implementor should remove all the information about the specified procId.
  
  Parameters:
  
  procId - the ID of the procedure to remove.
- delete
```
public void delete(Procedure<?> proc,
                   long[] subProcIds)
```
  Description copied from interface: ProcedureStore
  
  The parent procedure completed. Update the state and mark all the child deleted.
  
  Parameters:
  
  proc - the parent procedure to serialize and write to the store.
  
  subProcIds - the IDs of the sub-procedure to remove.
- delete
```
public void delete(long[] procIds,
                   int offset,
                   int count)
```
  Description copied from interface: ProcedureStore
  
  The specified procIds were removed from the executor, due to completion, abort or failure. The store implementor should remove all the information about the specified procIds.
  
  Parameters:
  
  procIds - the IDs of the procedures to remove.
  
  offset - the array offset from where to start to delete
  
  count - the number of IDs to delete
- delete
```
private void delete(long[] procIds)
```
- acquireSlot
```
private ByteSlot acquireSlot()
```
- releaseSlot
```
private void releaseSlot(ByteSlot slot)
```
- pushData
```
private long pushData(WALProcedureStore.PushType type,
                      ByteSlot slot,
                      long procId,
                      long[] subProcIds)
```
- updateStoreTracker
```
private void updateStoreTracker(WALProcedureStore.PushType type,
                                long procId,
                                long[] subProcIds)
```
- isSyncAborted
```
private boolean isSyncAborted()
```
- syncLoop
```
private void syncLoop()
               throws Throwable
```
  Throws:
  
  Throwable
- getSyncMetrics
```
public ArrayList<WALProcedureStore.SyncMetrics> getSyncMetrics()
```
- syncSlots
```
private long syncSlots()
                throws Throwable
```
  Throws:
  
  Throwable
- syncSlots
```
protected long syncSlots(org.apache.hadoop.fs.FSDataOutputStream stream,
                         ByteSlot[] slots,
                         int offset,
                         int count)
                  throws IOException
```
  Throws:
  
  IOException
- syncStream
```
protected void syncStream(org.apache.hadoop.fs.FSDataOutputStream stream)
                   throws IOException
```
  Throws:
  
  IOException
- rollWriterWithRetries
```
private boolean rollWriterWithRetries()
```
- tryRollWriter
```
private boolean tryRollWriter()
```
- getMillisToNextPeriodicRoll
```
public long getMillisToNextPeriodicRoll()
```
- getMillisFromLastRoll
```
public long getMillisFromLastRoll()
```
- periodicRollForTesting
```
void periodicRollForTesting()
                     throws IOException
```
  Throws:
  
  IOException
- rollWriterForTesting
```
public boolean rollWriterForTesting()
                             throws IOException
```
  Throws:
  
  IOException
- removeInactiveLogsForTesting
```
void removeInactiveLogsForTesting()
                           throws Exception
```
  Throws:
  
  Exception
- periodicRoll
```
private void periodicRoll()
                   throws IOException
```
  Throws:
  
  IOException
- rollWriter
```
private boolean rollWriter()
                    throws IOException
```
  Throws:
  
  IOException
- rollWriter
```
boolean rollWriter(long logId)
            throws IOException
```
  Throws:
  
  IOException
- closeCurrentLogStream
```
private void closeCurrentLogStream(boolean abort)
```
- removeInactiveLogs
```
private void removeInactiveLogs()
                         throws IOException
```
  Throws:
  
  IOException
- buildHoldingCleanupTracker
```
private void buildHoldingCleanupTracker()
```
- removeAllLogs
```
private void removeAllLogs(long lastLogId,
                           String why)
```
  Remove all logs with logId <= lastLogId.
- removeLogFile
```
private boolean removeLogFile(ProcedureWALFile log,
                              org.apache.hadoop.fs.Path walArchiveDir)
```
- getWALDir
```
public org.apache.hadoop.fs.Path getWALDir()
```
- getWalArchiveDir
```
org.apache.hadoop.fs.Path getWalArchiveDir()
```
- getFileSystem
```
public org.apache.hadoop.fs.FileSystem getFileSystem()
```
- getLogFilePath
```
protected org.apache.hadoop.fs.Path getLogFilePath(long logId)
                                            throws IOException
```
  Throws:
  
  IOException
- getLogIdFromName
```
private static long getLogIdFromName(String name)
```
- getLogFiles
```
private org.apache.hadoop.fs.FileStatus[] getLogFiles()
                                               throws IOException
```
  Throws:
  
  IOException
- getMaxLogId
```
private static long getMaxLogId(org.apache.hadoop.fs.FileStatus[] logFiles)
```
  Make sure that the file set are gotten by calling getLogFiles(), where we will sort the file set by log id.
  
  Returns:
  
  Max-LogID of the specified log file set
- initOldLogs
```
private long initOldLogs(org.apache.hadoop.fs.FileStatus[] logFiles)
                  throws IOException
```
  Make sure that the file set are gotten by calling getLogFiles(), where we will sort the file set by log id.
  
  Returns:
  
  Max-LogID of the specified log file set
  
  Throws:
  
  IOException
- initTrackerFromOldLogs
```
private void initTrackerFromOldLogs()
```
  If last log's tracker is not null, use it as storeTracker. Otherwise, set storeTracker as partial, and let ProcedureWALFormatReader rebuild it using entries in the log.
- initOldLog
```
private ProcedureWALFile initOldLog(org.apache.hadoop.fs.FileStatus logFile,
                                    org.apache.hadoop.fs.Path walArchiveDir)
                             throws IOException
```
  Loads given log file and it's tracker.
  
  Throws:
  
  IOException
- main
```
public static void main(String[] args)
                 throws IOException
```
  Parses a directory of WALs building up ProcedureState. For testing parse and profiling.
  
  Parameters:
  
  args - Include pointer to directory of WAL files for a store instance to parse & load.
  
  Throws:
  
  IOException

Class WALProcedureStore

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hadoop.hbase.procedure2.store.ProcedureStore

Field Summary

Constructor Summary

Method Summary

Methods inherited from class org.apache.hadoop.hbase.procedure2.store.ProcedureStoreBase

Methods inherited from class java.lang.Object

Field Detail

LOG

LOG_PREFIX

MASTER_PROCEDURE_LOGDIR

WAL_COUNT_WARN_THRESHOLD_CONF_KEY

DEFAULT_WAL_COUNT_WARN_THRESHOLD

EXEC_WAL_CLEANUP_ON_LOAD_CONF_KEY

DEFAULT_EXEC_WAL_CLEANUP_ON_LOAD_CONF_KEY

MAX_RETRIES_BEFORE_ROLL_CONF_KEY

DEFAULT_MAX_RETRIES_BEFORE_ROLL

WAIT_BEFORE_ROLL_CONF_KEY

DEFAULT_WAIT_BEFORE_ROLL

ROLL_RETRIES_CONF_KEY

DEFAULT_ROLL_RETRIES

MAX_SYNC_FAILURE_ROLL_CONF_KEY

DEFAULT_MAX_SYNC_FAILURE_ROLL

PERIODIC_ROLL_CONF_KEY

DEFAULT_PERIODIC_ROLL

SYNC_WAIT_MSEC_CONF_KEY

DEFAULT_SYNC_WAIT_MSEC

USE_HSYNC_CONF_KEY

DEFAULT_USE_HSYNC

ROLL_THRESHOLD_CONF_KEY

DEFAULT_ROLL_THRESHOLD

STORE_WAL_SYNC_STATS_COUNT

DEFAULT_SYNC_STATS_COUNT

logs

holdingCleanupTracker

storeTracker

lock

waitCond

slotCond

syncCond

leaseRecovery

conf

fs

walDir

walArchiveDir

enforceStreamCapability

syncException

loading

inSync

totalSynced

lastRollTs

syncId

slotsCache

corruptedLogs

stream

runningProcCount

flushLogId

syncMaxSlot

slotIndex

syncThread

slots

walCountWarnThreshold

maxRetriesBeforeRoll

maxSyncFailureRoll

waitBeforeRoll

rollRetries

periodicRollMsec

rollThreshold

useHsync

syncWaitMsec

syncMetricsQueue

WALS_PATH_FILTER

FILE_STATUS_ID_COMPARATOR

Constructor Detail

WALProcedureStore

WALProcedureStore

Method Detail

start

stop