CompactingMemStore (Apache HBase 2.5.0 API)

java.lang.Object
- org.apache.hadoop.hbase.regionserver.AbstractMemStore
- - org.apache.hadoop.hbase.regionserver.CompactingMemStore

All Implemented Interfaces:

MemStore
```
@InterfaceAudience.Private
public class CompactingMemStore
extends AbstractMemStore
```
A memstore implementation which supports in-memory compaction. A compaction pipeline is added between the active set and the snapshot data structures; it consists of a list of segments that are subject to compaction. Like the snapshot, all pipeline segments are read-only; updates only affect the active set. To ensure this property we take advantage of the existing blocking mechanism -- the active set is pushed to the pipeline while holding the region's updatesLock in exclusive mode. Periodically, a compaction is applied in the background to all pipeline segments resulting in a single read-only component. The ``old'' segments are discarded when no scanner is reading them.

Nested Class Summary

Nested Classes
Modifier and Type	Class and Description
`static class`	`CompactingMemStore.IndexType` Types of indexes (part of immutable segments) to be used after flattening, compaction, or merge are applied.
`private class`	`CompactingMemStore.InMemoryCompactionRunnable` The in-memory-flusher thread performs the flush asynchronously.

Field Summary

Fields
Modifier and Type	Field and Description
`protected AtomicBoolean`	`allowCompaction`
`static String`	`COMPACTING_MEMSTORE_TYPE_DEFAULT`
`static String`	`COMPACTING_MEMSTORE_TYPE_KEY`
`protected MemStoreCompactor`	`compactor`
`private boolean`	`compositeSnapshot`
`static long`	`DEEP_OVERHEAD`
`static int`	`IN_MEMORY_CONPACTION_POOL_SIZE_DEFAULT`
`static String`	`IN_MEMORY_CONPACTION_POOL_SIZE_KEY`
`private static int`	`IN_MEMORY_FLUSH_MULTIPLIER`
`static String`	`IN_MEMORY_FLUSH_THRESHOLD_FACTOR_KEY`
`private CompactingMemStore.IndexType`	`indexType`
`private AtomicBoolean`	`inMemoryCompactionInProgress`
`private long`	`inmemoryFlushSize`
`private boolean`	`inWalReplay`
`private static org.slf4j.Logger`	`LOG`
`private CompactionPipeline`	`pipeline`
`private HStore`	`store`

Fields inherited from class org.apache.hadoop.hbase.regionserver.AbstractMemStore
FIXED_OVERHEAD, regionServices, snapshot, snapshotId

Constructor Summary

Constructors
Constructor and Description
`CompactingMemStore(org.apache.hadoop.conf.Configuration conf, CellComparator c, HStore store, RegionServicesForStores regionServices, MemoryCompactionPolicy compactionPolicy)`

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`protected boolean`	`checkAndAddToActiveSize(MutableSegment currActive, Cell cellToAdd, MemStoreSizing memstoreSizing)` Check whether anything need to be done based on the current active set size.
`protected List<KeyValueScanner>`	`createList(int capacity)`
`protected MemStoreCompactor`	`createMemStoreCompactor(MemoryCompactionPolicy compactionPolicy)`
`void`	`debug()`
`void`	`flattenOneSegment(long requesterVersion, MemStoreCompactionStrategy.Action action)`
`(package private) void`	`flushInMemory()`
`protected void`	`flushInMemory(MutableSegment currActive)`
`String`	`getFamilyName()`
`private byte[]`	`getFamilyNameInBytes()`
`MemStoreSize`	`getFlushableSize()` Flush will first clear out the data in snapshot if any (It will take a second flush invocation to clear the current Cell set).
`VersionedSegmentsList`	`getImmutableSegments()`
`CompactingMemStore.IndexType`	`getIndexType()`
`(package private) long`	`getInmemoryFlushSize()`
`private Segment`	`getLastSegment()`
`(package private) Cell`	`getNextRow(Cell cell)`
`private ThreadPoolExecutor`	`getPool()`
`private RegionServicesForStores`	`getRegionServices()`
`List<KeyValueScanner>`	`getScanners(long readPt)` This method is protected under HStore read lock.
`protected List<Segment>`	`getSegments()` Returns an ordered list of segments from most recent to oldest in memstore
`long`	`getSmallestReadPoint()`
`HStore`	`getStore()`
`boolean`	`hasImmutableSegments()`
`protected long`	`heapSize()`
`private void`	`initInmemoryFlushSize(org.apache.hadoop.conf.Configuration conf)`
`(package private) void`	`inMemoryCompaction()`
`(package private) boolean`	`isMemStoreFlushingInMemory()`
`boolean`	`isSloppy()`
`protected long`	`keySize()` Returns The total size of cells in this memstore.
`protected void`	`postUpdate(MutableSegment currentActive)` Issue any post update synchronization and tests
`long`	`preFlushSeqIDEstimation()` This method is called before the flush is executed.
`protected boolean`	`preUpdate(MutableSegment currentActive, Cell cell, MemStoreSizing memstoreSizing)` Issue any synchronization and test needed before applying the update For compacting memstore this means checking the update can increase the size without overflow
`protected void`	`pushActiveToPipeline(MutableSegment currActive, boolean checkEmpty)` NOTE: When `flushInMemory(MutableSegment)` calls this method, due to concurrent writes and because we first add cell size to currActive.getDataSize and then actually add cell to currActive.cellSet, it is possible that currActive.getDataSize could not accommodate cellToAdd but currActive.cellSet is still empty if pending writes which not yet add cells to currActive.cellSet,so for `flushInMemory(MutableSegment)`,checkEmpty parameter is false.
`private void`	`pushPipelineToSnapshot()`
`private void`	`pushTailToSnapshot()`
`private void`	`pushToSnapshot(List<ImmutableSegment> segments)`
`void`	`setCompositeSnapshot(boolean useCompositeSnapshot)`
`(package private) void`	`setIndexType(CompactingMemStore.IndexType type)`
`void`	`setInMemoryCompactionCompleted()`
`protected boolean`	`setInMemoryCompactionFlag()`
`MemStoreSize`	`size()`
`protected boolean`	`sizeAddedPreOperation()`
`MemStoreSnapshot`	`snapshot()` Push the current active memstore segment into the pipeline and create a snapshot of the tail of current compaction pipeline Snapshot must be cleared by call to `AbstractMemStore.clearSnapshot(long)`.
`void`	`startReplayingFromWAL()` This message intends to inform the MemStore that next coming updates are going to be part of the replaying edits from WAL
`private void`	`stopCompaction()` The request to cancel the compaction asynchronous task (caused by in-memory flush) The compaction may still happen if the request was sent too late Non-blocking request
`void`	`stopReplayingFromWAL()` This message intends to inform the MemStore that the replaying edits from WAL are done
`boolean`	`swapCompactedSegments(VersionedSegmentsList versionedList, ImmutableSegment result, boolean merge)`
`protected boolean`	`swapPipelineWithNull(VersionedSegmentsList segments)`
`private void`	`tryFlushInMemoryAndCompactingAsync(MutableSegment currActive)` Try to flush the currActive in memory and submit the background `CompactingMemStore.InMemoryCompactionRunnable` to `RegionServicesForStores.getInMemoryCompactionPool()`.
`void`	`updateLowestUnflushedSequenceIdInWAL(boolean onlyIfGreater)` Updates the wal with the lowest sequence id (oldest entry) that is still in memory

Methods inherited from class org.apache.hadoop.hbase.regionserver.AbstractMemStore
add, add, addToScanners, addToScanners, clearSnapshot, doAdd, doClearSnapShot, dump, getActive, getComparator, getConfiguration, getLowest, getNextRow, getSnapshot, getSnapshotSize, resetActive, resetTimeOfOldestEdit, timeOfOldestEdit, toString, upsert

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

- Field Detail
  - COMPACTING_MEMSTORE_TYPE_KEY
```
public static final String COMPACTING_MEMSTORE_TYPE_KEY
```
    See Also:
    
    Constant Field Values
  - COMPACTING_MEMSTORE_TYPE_DEFAULT
```
public static final String COMPACTING_MEMSTORE_TYPE_DEFAULT
```
  - IN_MEMORY_FLUSH_THRESHOLD_FACTOR_KEY
```
public static final String IN_MEMORY_FLUSH_THRESHOLD_FACTOR_KEY
```
    See Also:
    
    Constant Field Values
  - IN_MEMORY_FLUSH_MULTIPLIER
```
private static final int IN_MEMORY_FLUSH_MULTIPLIER
```
    See Also:
    
    Constant Field Values
  - IN_MEMORY_CONPACTION_POOL_SIZE_KEY
```
public static final String IN_MEMORY_CONPACTION_POOL_SIZE_KEY
```
    See Also:
    
    Constant Field Values
  - IN_MEMORY_CONPACTION_POOL_SIZE_DEFAULT
```
public static final int IN_MEMORY_CONPACTION_POOL_SIZE_DEFAULT
```
    See Also:
    
    Constant Field Values
  - LOG
```
private static final org.slf4j.Logger LOG
```
  - store
```
private HStore store
```
  - pipeline
```
private CompactionPipeline pipeline
```
  - compactor
```
protected MemStoreCompactor compactor
```
  - inmemoryFlushSize
```
private long inmemoryFlushSize
```
  - inMemoryCompactionInProgress
```
private final AtomicBoolean inMemoryCompactionInProgress
```
  - inWalReplay
```
private boolean inWalReplay
```
  - allowCompaction
```
protected final AtomicBoolean allowCompaction
```
  - compositeSnapshot
```
private boolean compositeSnapshot
```
  - indexType
```
private CompactingMemStore.IndexType indexType
```
  - DEEP_OVERHEAD
```
public static final long DEEP_OVERHEAD
```
- Constructor Detail
  - CompactingMemStore
```
public CompactingMemStore(org.apache.hadoop.conf.Configuration conf,
                          CellComparator c,
                          HStore store,
                          RegionServicesForStores regionServices,
                          MemoryCompactionPolicy compactionPolicy)
                   throws IOException
```
    Throws:
    
    IOException
- Method Detail
  - createMemStoreCompactor
```
protected MemStoreCompactor createMemStoreCompactor(MemoryCompactionPolicy compactionPolicy)
                                             throws IllegalArgumentIOException
```
    Throws:
    
    IllegalArgumentIOException
  - initInmemoryFlushSize
```
private void initInmemoryFlushSize(org.apache.hadoop.conf.Configuration conf)
```
  - size
```
public MemStoreSize size()
```
    Returns:
    
    Total memory occupied by this MemStore. This won't include any size occupied by the snapshot. We assume the snapshot will get cleared soon. This is not thread safe and the memstore may be changed while computing its size. It is the responsibility of the caller to make sure this doesn't happen.
  - preFlushSeqIDEstimation
```
public long preFlushSeqIDEstimation()
```
    This method is called before the flush is executed.
    
    Returns:
    
    an estimation (lower bound) of the unflushed sequence id in memstore after the flush is executed. if memstore will be cleared returns HConstants.NO_SEQNUM.
  - isSloppy
```
public boolean isSloppy()
```
  - snapshot
```
public MemStoreSnapshot snapshot()
```
    Push the current active memstore segment into the pipeline and create a snapshot of the tail of current compaction pipeline Snapshot must be cleared by call to AbstractMemStore.clearSnapshot(long). AbstractMemStore.clearSnapshot(long).
    
    Returns:
    
    MemStoreSnapshot
  - getFlushableSize
```
public MemStoreSize getFlushableSize()
```
    Description copied from interface: MemStore
    
    Flush will first clear out the data in snapshot if any (It will take a second flush invocation to clear the current Cell set). If snapshot is empty, current Cell set will be flushed.
    
    Returns:
    
    On flush, how much memory we will clear.
  - setInMemoryCompactionCompleted
```
public void setInMemoryCompactionCompleted()
```
  - setInMemoryCompactionFlag
```
protected boolean setInMemoryCompactionFlag()
```
  - keySize
```
protected long keySize()
```
    Description copied from class: AbstractMemStore
    
    Returns The total size of cells in this memstore. We will not consider cells in the snapshot
    
    Specified by:
    
    keySize in class AbstractMemStore
  - heapSize
```
protected long heapSize()
```
    Specified by:
    
    heapSize in class AbstractMemStore
    
    Returns:
    
    The total heap size of cells in this memstore. We will not consider cells in the snapshot
  - updateLowestUnflushedSequenceIdInWAL
```
public void updateLowestUnflushedSequenceIdInWAL(boolean onlyIfGreater)
```
    Description copied from class: AbstractMemStore
    
    Updates the wal with the lowest sequence id (oldest entry) that is still in memory
    
    Specified by:
    
    updateLowestUnflushedSequenceIdInWAL in class AbstractMemStore
    
    Parameters:
    
    onlyIfGreater - a flag that marks whether to update the sequence id no matter what or only if it is greater than the previous sequence id
  - startReplayingFromWAL
```
public void startReplayingFromWAL()
```
    This message intends to inform the MemStore that next coming updates are going to be part of the replaying edits from WAL
  - stopReplayingFromWAL
```
public void stopReplayingFromWAL()
```
    This message intends to inform the MemStore that the replaying edits from WAL are done
  - preUpdate
```
protected boolean preUpdate(MutableSegment currentActive,
                            Cell cell,
                            MemStoreSizing memstoreSizing)
```
    Issue any synchronization and test needed before applying the update For compacting memstore this means checking the update can increase the size without overflow
    
    Specified by:
    
    preUpdate in class AbstractMemStore
    
    Parameters:
    
    currentActive - the segment to be updated
    
    cell - the cell to be added
    
    memstoreSizing - object to accumulate region size changes
    
    Returns:
    
    true iff can proceed with applying the update
  - postUpdate
```
protected void postUpdate(MutableSegment currentActive)
```
    Description copied from class: AbstractMemStore
    
    Issue any post update synchronization and tests
    
    Specified by:
    
    postUpdate in class AbstractMemStore
    
    Parameters:
    
    currentActive - updated segment
  - sizeAddedPreOperation
```
protected boolean sizeAddedPreOperation()
```
    Specified by:
    
    sizeAddedPreOperation in class AbstractMemStore
  - getSegments
```
protected List<Segment> getSegments()
```
    Description copied from class: AbstractMemStore
    
    Returns an ordered list of segments from most recent to oldest in memstore
    
    Specified by:
    
    getSegments in class AbstractMemStore
  - setCompositeSnapshot
```
public void setCompositeSnapshot(boolean useCompositeSnapshot)
```
  - swapCompactedSegments
```
public boolean swapCompactedSegments(VersionedSegmentsList versionedList,
                                     ImmutableSegment result,
                                     boolean merge)
```
  - flattenOneSegment
```
public void flattenOneSegment(long requesterVersion,
                              MemStoreCompactionStrategy.Action action)
```
    Parameters:
    
    requesterVersion - The caller must hold the VersionedList of the pipeline with version taken earlier. This version must be passed as a parameter here. The flattening happens only if versions match.
  - setIndexType
```
void setIndexType(CompactingMemStore.IndexType type)
```
  - getIndexType
```
public CompactingMemStore.IndexType getIndexType()
```
  - hasImmutableSegments
```
public boolean hasImmutableSegments()
```
  - getImmutableSegments
```
public VersionedSegmentsList getImmutableSegments()
```
  - getSmallestReadPoint
```
public long getSmallestReadPoint()
```
  - getStore
```
public HStore getStore()
```
  - getFamilyName
```
public String getFamilyName()
```
  - getScanners
```
public List<KeyValueScanner> getScanners(long readPt)
                                  throws IOException
```
    This method is protected under HStore read lock.
    
    Returns:
    
    scanner over the memstore. This might include scanner over the snapshot when one is present.
    
    Throws:
    
    IOException
  - createList
```
protected List<KeyValueScanner> createList(int capacity)
```
  - checkAndAddToActiveSize
```
protected boolean checkAndAddToActiveSize(MutableSegment currActive,
                                          Cell cellToAdd,
                                          MemStoreSizing memstoreSizing)
```
    Check whether anything need to be done based on the current active set size. The method is invoked upon every addition to the active set. For CompactingMemStore, flush the active set to the read-only memory if it's size is above threshold
    
    Parameters:
    
    currActive - intended segment to update
    
    cellToAdd - cell to be added to the segment
    
    memstoreSizing - object to accumulate changed size
    
    Returns:
    
    true if the cell can be added to the currActive
  - tryFlushInMemoryAndCompactingAsync
```
private void tryFlushInMemoryAndCompactingAsync(MutableSegment currActive)
```
    Try to flush the currActive in memory and submit the background CompactingMemStore.InMemoryCompactionRunnable to RegionServicesForStores.getInMemoryCompactionPool(). Just one thread can do the actual flushing in memory.
    
    Parameters:
    
    currActive - current Active Segment to be flush in memory.
  - flushInMemory
```
void flushInMemory()
```
  - flushInMemory
```
protected void flushInMemory(MutableSegment currActive)
```
  - inMemoryCompaction
```
void inMemoryCompaction()
```
  - getLastSegment
```
private Segment getLastSegment()
```
  - getFamilyNameInBytes
```
private byte[] getFamilyNameInBytes()
```
  - getPool
```
private ThreadPoolExecutor getPool()
```
  - stopCompaction
```
private void stopCompaction()
```
    The request to cancel the compaction asynchronous task (caused by in-memory flush) The compaction may still happen if the request was sent too late Non-blocking request
  - pushActiveToPipeline
```
protected void pushActiveToPipeline(MutableSegment currActive,
                                    boolean checkEmpty)
```
    NOTE: When flushInMemory(MutableSegment) calls this method, due to concurrent writes and because we first add cell size to currActive.getDataSize and then actually add cell to currActive.cellSet, it is possible that currActive.getDataSize could not accommodate cellToAdd but currActive.cellSet is still empty if pending writes which not yet add cells to currActive.cellSet,so for flushInMemory(MutableSegment),checkEmpty parameter is false. But if snapshot() called this method,because there is no pending write,checkEmpty parameter could be true. nn
  - pushTailToSnapshot
```
private void pushTailToSnapshot()
```
  - pushPipelineToSnapshot
```
private void pushPipelineToSnapshot()
```
  - swapPipelineWithNull
```
protected boolean swapPipelineWithNull(VersionedSegmentsList segments)
```
  - pushToSnapshot
```
private void pushToSnapshot(List<ImmutableSegment> segments)
```
  - getRegionServices
```
private RegionServicesForStores getRegionServices()
```
  - isMemStoreFlushingInMemory
```
boolean isMemStoreFlushingInMemory()
```
  - getNextRow
```
Cell getNextRow(Cell cell)
```
    Parameters:
    
    cell - Find the row that comes after this one. If null, we return the first.
    
    Returns:
    
    Next row or null if none found.
  - getInmemoryFlushSize
```
long getInmemoryFlushSize()
```
  - debug
```
public void debug()
```

Class CompactingMemStore

Nested Class Summary

Field Summary

Fields inherited from class org.apache.hadoop.hbase.regionserver.AbstractMemStore

Constructor Summary

Method Summary

Methods inherited from class org.apache.hadoop.hbase.regionserver.AbstractMemStore

Methods inherited from class java.lang.Object

Field Detail

COMPACTING_MEMSTORE_TYPE_KEY

COMPACTING_MEMSTORE_TYPE_DEFAULT

IN_MEMORY_FLUSH_THRESHOLD_FACTOR_KEY

IN_MEMORY_FLUSH_MULTIPLIER

IN_MEMORY_CONPACTION_POOL_SIZE_KEY

IN_MEMORY_CONPACTION_POOL_SIZE_DEFAULT

LOG

store

pipeline

compactor

inmemoryFlushSize

inMemoryCompactionInProgress

inWalReplay

allowCompaction

compositeSnapshot

indexType

DEEP_OVERHEAD

Constructor Detail

CompactingMemStore

Method Detail

createMemStoreCompactor

initInmemoryFlushSize

size

preFlushSeqIDEstimation

isSloppy

snapshot

getFlushableSize

setInMemoryCompactionCompleted

setInMemoryCompactionFlag

keySize

heapSize

updateLowestUnflushedSequenceIdInWAL

startReplayingFromWAL

stopReplayingFromWAL

preUpdate

postUpdate

sizeAddedPreOperation

getSegments

setCompositeSnapshot

swapCompactedSegments

flattenOneSegment

setIndexType

getIndexType

hasImmutableSegments

getImmutableSegments

getSmallestReadPoint

getStore

getFamilyName

getScanners

createList

checkAndAddToActiveSize

tryFlushInMemoryAndCompactingAsync

flushInMemory

flushInMemory

inMemoryCompaction

getLastSegment

getFamilyNameInBytes

getPool

stopCompaction

pushActiveToPipeline

pushTailToSnapshot

pushPipelineToSnapshot

swapPipelineWithNull

pushToSnapshot

getRegionServices

isMemStoreFlushingInMemory

getNextRow

getInmemoryFlushSize

debug