CompactingMemStore (Apache HBase 2.0.6 API)

java.lang.Object
- org.apache.hadoop.hbase.regionserver.AbstractMemStore
- - org.apache.hadoop.hbase.regionserver.CompactingMemStore

All Implemented Interfaces:

MemStore
```
@InterfaceAudience.Private
public class CompactingMemStore
extends AbstractMemStore
```
A memstore implementation which supports in-memory compaction. A compaction pipeline is added between the active set and the snapshot data structures; it consists of a list of segments that are subject to compaction. Like the snapshot, all pipeline segments are read-only; updates only affect the active set. To ensure this property we take advantage of the existing blocking mechanism -- the active set is pushed to the pipeline while holding the region's updatesLock in exclusive mode. Periodically, a compaction is applied in the background to all pipeline segments resulting in a single read-only component. The ``old'' segments are discarded when no scanner is reading them.

Nested Class Summary

Nested Classes
Modifier and Type	Class and Description
`static class`	`CompactingMemStore.IndexType` Types of indexes (part of immutable segments) to be used after flattening, compaction, or merge are applied.
`private class`	`CompactingMemStore.InMemoryFlushRunnable` The in-memory-flusher thread performs the flush asynchronously.

Field Summary

Fields
Modifier and Type	Field and Description
`protected AtomicBoolean`	`allowCompaction`
`static String`	`COMPACTING_MEMSTORE_TYPE_DEFAULT`
`static String`	`COMPACTING_MEMSTORE_TYPE_KEY`
`protected MemStoreCompactor`	`compactor`
`private boolean`	`compositeSnapshot`
`static long`	`DEEP_OVERHEAD`
`static int`	`IN_MEMORY_CONPACTION_POOL_SIZE_DEFAULT`
`static String`	`IN_MEMORY_CONPACTION_POOL_SIZE_KEY`
`private static double`	`IN_MEMORY_FLUSH_THRESHOLD_FACTOR_DEFAULT`
`static String`	`IN_MEMORY_FLUSH_THRESHOLD_FACTOR_KEY`
`private CompactingMemStore.IndexType`	`indexType`
`private AtomicBoolean`	`inMemoryFlushInProgress`
`private long`	`inmemoryFlushSize`
`private boolean`	`inWalReplay`
`private static org.slf4j.Logger`	`LOG`
`private CompactionPipeline`	`pipeline`
`private HStore`	`store`

Fields inherited from class org.apache.hadoop.hbase.regionserver.AbstractMemStore
active, FIXED_OVERHEAD, regionServices, snapshot, snapshotId

Constructor Summary

Constructors
Constructor and Description
`CompactingMemStore(org.apache.hadoop.conf.Configuration conf, CellComparator c, HStore store, RegionServicesForStores regionServices, MemoryCompactionPolicy compactionPolicy)`

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`protected void`	`checkActiveSize()` Check whether anything need to be done based on the current active set size.
`protected List<KeyValueScanner>`	`createList(int capacity)`
`protected MemStoreCompactor`	`createMemStoreCompactor(MemoryCompactionPolicy compactionPolicy)`
`void`	`debug()`
`void`	`flattenOneSegment(long requesterVersion, MemStoreCompactionStrategy.Action action)`
`(package private) void`	`flushInMemory()`
`String`	`getFamilyName()`
`private byte[]`	`getFamilyNameInBytes()`
`MemStoreSize`	`getFlushableSize()` Flush will first clear out the data in snapshot if any (It will take a second flush invocation to clear the current Cell set).
`VersionedSegmentsList`	`getImmutableSegments()`
`CompactingMemStore.IndexType`	`getIndexType()`
`(package private) long`	`getInmemoryFlushSize()`
`private Segment`	`getLastSegment()`
`(package private) Cell`	`getNextRow(Cell cell)`
`private ThreadPoolExecutor`	`getPool()`
`private RegionServicesForStores`	`getRegionServices()`
`List<KeyValueScanner>`	`getScanners(long readPt)`
`protected List<Segment>`	`getSegments()`
`long`	`getSmallestReadPoint()`
`HStore`	`getStore()`
`boolean`	`hasImmutableSegments()`
`protected long`	`heapSize()`
`private void`	`initInmemoryFlushSize(org.apache.hadoop.conf.Configuration conf)`
`(package private) boolean`	`isMemStoreFlushingInMemory()`
`boolean`	`isSloppy()`
`protected long`	`keySize()`
`long`	`preFlushSeqIDEstimation()` This method is called before the flush is executed.
`protected void`	`pushActiveToPipeline(MutableSegment active)`
`private void`	`pushPipelineToSnapshot()`
`private void`	`pushTailToSnapshot()`
`private void`	`pushToSnapshot(List<ImmutableSegment> segments)`
`void`	`setCompositeSnapshot(boolean useCompositeSnapshot)`
`(package private) void`	`setIndexType(CompactingMemStore.IndexType type)`
`protected boolean`	`shouldFlushInMemory()`
`MemStoreSize`	`size()`
`MemStoreSnapshot`	`snapshot()` Push the current active memstore segment into the pipeline and create a snapshot of the tail of current compaction pipeline Snapshot must be cleared by call to `AbstractMemStore.clearSnapshot(long)`.
`void`	`startReplayingFromWAL()` This message intends to inform the MemStore that next coming updates are going to be part of the replaying edits from WAL
`private void`	`stopCompaction()` The request to cancel the compaction asynchronous task (caused by in-memory flush) The compaction may still happen if the request was sent too late Non-blocking request
`void`	`stopReplayingFromWAL()` This message intends to inform the MemStore that the replaying edits from WAL are done
`boolean`	`swapCompactedSegments(VersionedSegmentsList versionedList, ImmutableSegment result, boolean merge)`
`void`	`updateLowestUnflushedSequenceIdInWAL(boolean onlyIfGreater)` Updates the wal with the lowest sequence id (oldest entry) that is still in memory

Methods inherited from class org.apache.hadoop.hbase.regionserver.AbstractMemStore
add, add, addToScanners, addToScanners, clearSnapshot, dump, getActive, getComparator, getConfiguration, getLowest, getNextRow, getSnapshot, getSnapshotSize, resetActive, timeOfOldestEdit, toString, upsert

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

- Field Detail
  - COMPACTING_MEMSTORE_TYPE_KEY
```
public static final String COMPACTING_MEMSTORE_TYPE_KEY
```
    See Also:
    
    Constant Field Values
  - COMPACTING_MEMSTORE_TYPE_DEFAULT
```
public static final String COMPACTING_MEMSTORE_TYPE_DEFAULT
```
  - IN_MEMORY_FLUSH_THRESHOLD_FACTOR_KEY
```
public static final String IN_MEMORY_FLUSH_THRESHOLD_FACTOR_KEY
```
    See Also:
    
    Constant Field Values
  - IN_MEMORY_FLUSH_THRESHOLD_FACTOR_DEFAULT
```
private static final double IN_MEMORY_FLUSH_THRESHOLD_FACTOR_DEFAULT
```
    See Also:
    
    Constant Field Values
  - IN_MEMORY_CONPACTION_POOL_SIZE_KEY
```
public static final String IN_MEMORY_CONPACTION_POOL_SIZE_KEY
```
    See Also:
    
    Constant Field Values
  - IN_MEMORY_CONPACTION_POOL_SIZE_DEFAULT
```
public static final int IN_MEMORY_CONPACTION_POOL_SIZE_DEFAULT
```
    See Also:
    
    Constant Field Values
  - LOG
```
private static final org.slf4j.Logger LOG
```
  - store
```
private HStore store
```
  - pipeline
```
private CompactionPipeline pipeline
```
  - compactor
```
protected MemStoreCompactor compactor
```
  - inmemoryFlushSize
```
private long inmemoryFlushSize
```
  - inMemoryFlushInProgress
```
private final AtomicBoolean inMemoryFlushInProgress
```
  - inWalReplay
```
private boolean inWalReplay
```
  - allowCompaction
```
protected final AtomicBoolean allowCompaction
```
  - compositeSnapshot
```
private boolean compositeSnapshot
```
  - indexType
```
private CompactingMemStore.IndexType indexType
```
  - DEEP_OVERHEAD
```
public static final long DEEP_OVERHEAD
```
- Constructor Detail
  - CompactingMemStore
```
public CompactingMemStore(org.apache.hadoop.conf.Configuration conf,
                          CellComparator c,
                          HStore store,
                          RegionServicesForStores regionServices,
                          MemoryCompactionPolicy compactionPolicy)
                   throws IOException
```
    Throws:
    
    IOException
- Method Detail
  - createMemStoreCompactor
```
protected MemStoreCompactor createMemStoreCompactor(MemoryCompactionPolicy compactionPolicy)
                                             throws IllegalArgumentIOException
```
    Throws:
    
    IllegalArgumentIOException
  - initInmemoryFlushSize
```
private void initInmemoryFlushSize(org.apache.hadoop.conf.Configuration conf)
```
  - size
```
public MemStoreSize size()
```
    Returns:
    
    Total memory occupied by this MemStore. This won't include any size occupied by the snapshot. We assume the snapshot will get cleared soon. This is not thread safe and the memstore may be changed while computing its size. It is the responsibility of the caller to make sure this doesn't happen.
  - preFlushSeqIDEstimation
```
public long preFlushSeqIDEstimation()
```
    This method is called before the flush is executed.
    
    Returns:
    
    an estimation (lower bound) of the unflushed sequence id in memstore after the flush is executed. if memstore will be cleared returns HConstants.NO_SEQNUM.
  - isSloppy
```
public boolean isSloppy()
```
  - snapshot
```
public MemStoreSnapshot snapshot()
```
    Push the current active memstore segment into the pipeline and create a snapshot of the tail of current compaction pipeline Snapshot must be cleared by call to AbstractMemStore.clearSnapshot(long). AbstractMemStore.clearSnapshot(long).
    
    Returns:
    
    MemStoreSnapshot
  - getFlushableSize
```
public MemStoreSize getFlushableSize()
```
    Description copied from interface: MemStore
    
    Flush will first clear out the data in snapshot if any (It will take a second flush invocation to clear the current Cell set). If snapshot is empty, current Cell set will be flushed.
    
    Returns:
    
    On flush, how much memory we will clear.
  - keySize
```
protected long keySize()
```
    Specified by:
    
    keySize in class AbstractMemStore
    
    Returns:
    
    The total size of cells in this memstore. We will not consider cells in the snapshot
  - heapSize
```
protected long heapSize()
```
    Specified by:
    
    heapSize in class AbstractMemStore
    
    Returns:
    
    The total heap size of cells in this memstore. We will not consider cells in the snapshot
  - updateLowestUnflushedSequenceIdInWAL
```
public void updateLowestUnflushedSequenceIdInWAL(boolean onlyIfGreater)
```
    Description copied from class: AbstractMemStore
    
    Updates the wal with the lowest sequence id (oldest entry) that is still in memory
    
    Specified by:
    
    updateLowestUnflushedSequenceIdInWAL in class AbstractMemStore
    
    Parameters:
    
    onlyIfGreater - a flag that marks whether to update the sequence id no matter what or only if it is greater than the previous sequence id
  - startReplayingFromWAL
```
public void startReplayingFromWAL()
```
    This message intends to inform the MemStore that next coming updates are going to be part of the replaying edits from WAL
  - stopReplayingFromWAL
```
public void stopReplayingFromWAL()
```
    This message intends to inform the MemStore that the replaying edits from WAL are done
  - getSegments
```
protected List<Segment> getSegments()
```
    Specified by:
    
    getSegments in class AbstractMemStore
    
    Returns:
    
    an ordered list of segments from most recent to oldest in memstore
  - setCompositeSnapshot
```
public void setCompositeSnapshot(boolean useCompositeSnapshot)
```
  - swapCompactedSegments
```
public boolean swapCompactedSegments(VersionedSegmentsList versionedList,
                                     ImmutableSegment result,
                                     boolean merge)
```
  - flattenOneSegment
```
public void flattenOneSegment(long requesterVersion,
                              MemStoreCompactionStrategy.Action action)
```
    Parameters:
    
    requesterVersion - The caller must hold the VersionedList of the pipeline with version taken earlier. This version must be passed as a parameter here. The flattening happens only if versions match.
  - setIndexType
```
void setIndexType(CompactingMemStore.IndexType type)
```
  - getIndexType
```
public CompactingMemStore.IndexType getIndexType()
```
  - hasImmutableSegments
```
public boolean hasImmutableSegments()
```
  - getImmutableSegments
```
public VersionedSegmentsList getImmutableSegments()
```
  - getSmallestReadPoint
```
public long getSmallestReadPoint()
```
  - getStore
```
public HStore getStore()
```
  - getFamilyName
```
public String getFamilyName()
```
  - getScanners
```
public List<KeyValueScanner> getScanners(long readPt)
                                  throws IOException
```
    Returns:
    
    scanner over the memstore. This might include scanner over the snapshot when one is present.
    
    Throws:
    
    IOException
  - createList
```
protected List<KeyValueScanner> createList(int capacity)
```
  - checkActiveSize
```
protected void checkActiveSize()
```
    Check whether anything need to be done based on the current active set size. The method is invoked upon every addition to the active set. For CompactingMemStore, flush the active set to the read-only memory if it's size is above threshold
    
    Specified by:
    
    checkActiveSize in class AbstractMemStore
  - flushInMemory
```
void flushInMemory()
            throws IOException
```
    Throws:
    
    IOException
  - getLastSegment
```
private Segment getLastSegment()
```
  - getFamilyNameInBytes
```
private byte[] getFamilyNameInBytes()
```
  - getPool
```
private ThreadPoolExecutor getPool()
```
  - shouldFlushInMemory
```
protected boolean shouldFlushInMemory()
```
  - stopCompaction
```
private void stopCompaction()
```
    The request to cancel the compaction asynchronous task (caused by in-memory flush) The compaction may still happen if the request was sent too late Non-blocking request
  - pushActiveToPipeline
```
protected void pushActiveToPipeline(MutableSegment active)
```
  - pushTailToSnapshot
```
private void pushTailToSnapshot()
```
  - pushPipelineToSnapshot
```
private void pushPipelineToSnapshot()
```
  - pushToSnapshot
```
private void pushToSnapshot(List<ImmutableSegment> segments)
```
  - getRegionServices
```
private RegionServicesForStores getRegionServices()
```
  - isMemStoreFlushingInMemory
```
boolean isMemStoreFlushingInMemory()
```
  - getNextRow
```
Cell getNextRow(Cell cell)
```
    Parameters:
    
    cell - Find the row that comes after this one. If null, we return the first.
    
    Returns:
    
    Next row or null if none found.
  - getInmemoryFlushSize
```
long getInmemoryFlushSize()
```
  - debug
```
public void debug()
```

Class CompactingMemStore

Nested Class Summary

Field Summary

Fields inherited from class org.apache.hadoop.hbase.regionserver.AbstractMemStore

Constructor Summary

Method Summary

Methods inherited from class org.apache.hadoop.hbase.regionserver.AbstractMemStore

Methods inherited from class java.lang.Object

Field Detail

COMPACTING_MEMSTORE_TYPE_KEY

COMPACTING_MEMSTORE_TYPE_DEFAULT

IN_MEMORY_FLUSH_THRESHOLD_FACTOR_KEY

IN_MEMORY_FLUSH_THRESHOLD_FACTOR_DEFAULT

IN_MEMORY_CONPACTION_POOL_SIZE_KEY

IN_MEMORY_CONPACTION_POOL_SIZE_DEFAULT

LOG

store

pipeline

compactor

inmemoryFlushSize

inMemoryFlushInProgress

inWalReplay

allowCompaction

compositeSnapshot

indexType

DEEP_OVERHEAD

Constructor Detail

CompactingMemStore

Method Detail

createMemStoreCompactor

initInmemoryFlushSize

size

preFlushSeqIDEstimation

isSloppy

snapshot

getFlushableSize

keySize

heapSize

updateLowestUnflushedSequenceIdInWAL

startReplayingFromWAL

stopReplayingFromWAL

getSegments

setCompositeSnapshot

swapCompactedSegments

flattenOneSegment

setIndexType

getIndexType

hasImmutableSegments

getImmutableSegments

getSmallestReadPoint

getStore

getFamilyName

getScanners

createList

checkActiveSize

flushInMemory

getLastSegment

getFamilyNameInBytes

getPool

shouldFlushInMemory

stopCompaction

pushActiveToPipeline

pushTailToSnapshot

pushPipelineToSnapshot

pushToSnapshot

getRegionServices

isMemStoreFlushingInMemory

getNextRow

getInmemoryFlushSize

debug