DefaultMemStore (Apache HBase 1.2.12 API)

java.lang.Object
- org.apache.hadoop.hbase.regionserver.DefaultMemStore

All Implemented Interfaces:

HeapSize, MemStore
```
@InterfaceAudience.Private
public class DefaultMemStore
extends Object
implements MemStore
```
The MemStore holds in-memory modifications to the Store. Modifications are Cells. When asked to flush, current memstore is moved to snapshot and is cleared. We continue to serve edits out of new memstore and backing snapshot until flusher reports in that the flush succeeded. At this point we let the snapshot go.
The MemStore functions should not be called in parallel. Callers should hold write and read locks. This is done in HStore.
TODO: Adjust size of the memstore when we remove items because they have been deleted. TODO: With new KVSLS, need to make sure we update HeapSize with difference in KV size.

Nested Class Summary

Nested Classes
Modifier and Type Class and Description

private static class DefaultMemStore.Member

protected class DefaultMemStore.MemStoreScanner

Nested Classes
Modifier and Type	Class and Description
`private static class`	`DefaultMemStore.Member`
`protected class`	`DefaultMemStore.MemStoreScanner`

Field Summary

Fields
Modifier and Type	Field and Description
`(package private) MemStoreLAB`	`allocator`
`(package private) CellSkipListSet`	`cellSet`
`(package private) KeyValue.KVComparator`	`comparator`
`private org.apache.hadoop.conf.Configuration`	`conf`
`static long`	`DEEP_OVERHEAD`
`static long`	`FIXED_OVERHEAD`
`private static org.apache.commons.logging.Log`	`LOG`
`(package private) static String`	`MSLAB_CLASS_NAME`
`(package private) AtomicLong`	`size`
`(package private) CellSkipListSet`	`snapshot`
`(package private) MemStoreLAB`	`snapshotAllocator`
`(package private) long`	`snapshotId`
`private long`	`snapshotSize`
`(package private) TimeRangeTracker`	`snapshotTimeRangeTracker`
`(package private) boolean`	`tagsPresent`
`(package private) long`	`timeOfOldestEdit`
`(package private) TimeRangeTracker`	`timeRangeTracker`
`private static boolean`	`USEMSLAB_DEFAULT`
`(package private) static String`	`USEMSLAB_KEY`

Constructor Summary

Constructors
Constructor and Description
`DefaultMemStore()` Default constructor.
`DefaultMemStore(org.apache.hadoop.conf.Configuration conf, KeyValue.KVComparator c)` Constructor.

Method Summary

Methods
Modifier and Type	Method and Description
`long`	`add(Cell cell)` Write an update
`private boolean`	`addToCellSet(Cell e)`
`void`	`clearSnapshot(long id)` The passed snapshot was successfully persisted; it can be let go.
`long`	`delete(Cell deleteCell)` Write a delete
`(package private) void`	`dump()`
`(package private) int`	`getCellLength(Cell cell)` Get cell length after serialized in `KeyValue`
`long`	`getFlushableSize()` On flush, how much memory we will clear.
`private Cell`	`getLowest(Cell a, Cell b)`
`(package private) Cell`	`getNextRow(Cell cell)`
`private Cell`	`getNextRow(Cell key, NavigableSet<Cell> set)`
`void`	`getRowKeyAtOrBefore(GetClosestRowBeforeTracker state)` Find the key that matches row exactly, or the one that immediately precedes it.
`private void`	`getRowKeyAtOrBefore(NavigableSet<Cell> set, GetClosestRowBeforeTracker state)`
`private void`	`getRowKeyBefore(NavigableSet<Cell> set, GetClosestRowBeforeTracker state)`
`List<KeyValueScanner>`	`getScanners(long readPt)`
`long`	`getSnapshotSize()` Return the size of the snapshot(s) if any
`long`	`heapSize()` Get the entire heap usage for this MemStore not including keys in the snapshot.
`(package private) static long`	`heapSizeChange(Cell cell, boolean notpresent)`
`private long`	`internalAdd(Cell toAdd, boolean mslabUsed)` Internal version of add() that doesn't clone Cells with the allocator, and doesn't take the lock.
`private long`	`keySize()`
`static void`	`main(String[] args)` Code to help figure if our approximation of object heap sizes is close enough.
`private Cell`	`maybeCloneWithAllocator(Cell cell)`
`private DefaultMemStore.Member`	`memberOfPreviousRow(NavigableSet<Cell> set, GetClosestRowBeforeTracker state, Cell firstOnRow)`
`private boolean`	`removeFromCellSet(Cell e)`
`void`	`rollback(Cell cell)` Remove n key from the memstore.
`(package private) void`	`setOldestEditTimeToNow()`
`boolean`	`shouldSeek(Scan scan, Store store, long oldestUnexpiredTS)` Check if this memstore may contain the required keys
`long`	`size()`
`MemStoreSnapshot`	`snapshot()` Creates a snapshot of the current memstore.
`long`	`timeOfOldestEdit()`
`long`	`updateColumnValue(byte[] row, byte[] family, byte[] qualifier, long newValue, long now)` Only used by tests.
`private long`	`upsert(Cell cell, long readpoint)` Inserts the specified KeyValue into MemStore and deletes any existing versions of the same row/family/qualifier as the specified KeyValue.
`long`	`upsert(Iterable<Cell> cells, long readpoint)` Update or insert the specified KeyValues.
`private boolean`	`walkForwardInSingleRow(SortedSet<Cell> set, Cell firstOnRow, GetClosestRowBeforeTracker state)`

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - LOG
```
private static final org.apache.commons.logging.Log LOG
```
  - USEMSLAB_KEY
```
static final String USEMSLAB_KEY
```
    See Also:
    Constant Field Values
  - USEMSLAB_DEFAULT
```
private static final boolean USEMSLAB_DEFAULT
```
    See Also:
    Constant Field Values
  - MSLAB_CLASS_NAME
```
static final String MSLAB_CLASS_NAME
```
    See Also:
    Constant Field Values
  - conf
```
private org.apache.hadoop.conf.Configuration conf
```
  - cellSet
```
volatile CellSkipListSet cellSet
```
  - snapshot
```
volatile CellSkipListSet snapshot
```
  - comparator
```
final KeyValue.KVComparator comparator
```
  - size
```
final AtomicLong size
```
  - snapshotSize
```
private volatile long snapshotSize
```
  - timeOfOldestEdit
```
volatile long timeOfOldestEdit
```
  - timeRangeTracker
```
TimeRangeTracker timeRangeTracker
```
  - snapshotTimeRangeTracker
```
TimeRangeTracker snapshotTimeRangeTracker
```
  - allocator
```
volatile MemStoreLAB allocator
```
  - snapshotAllocator
```
volatile MemStoreLAB snapshotAllocator
```
  - snapshotId
```
volatile long snapshotId
```
  - tagsPresent
```
volatile boolean tagsPresent
```
  - FIXED_OVERHEAD
```
public static final long FIXED_OVERHEAD
```
  - DEEP_OVERHEAD
```
public static final long DEEP_OVERHEAD
```
- Constructor Detail
  - DefaultMemStore
```
public DefaultMemStore()
```
    Default constructor. Used for tests.
  - DefaultMemStore
```
public DefaultMemStore(org.apache.hadoop.conf.Configuration conf,
               KeyValue.KVComparator c)
```
    Constructor.
    
    Parameters:
    c - Comparator
- Method Detail
  - dump
```
void dump()
```
  - snapshot
```
public MemStoreSnapshot snapshot()
```
    Creates a snapshot of the current memstore. Snapshot must be cleared by call to clearSnapshot(long)
    
    Specified by:
    
    snapshot in interface MemStore
    
    Returns:
    MemStoreSnapshot
  - clearSnapshot
```
public void clearSnapshot(long id)
                   throws UnexpectedStateException
```
    The passed snapshot was successfully persisted; it can be let go.
    
    Specified by:
    
    clearSnapshot in interface MemStore
    
    Parameters:
    id - Id of the snapshot to clean out.
    
    Throws:
    
    UnexpectedStateException
    See Also:
    snapshot()
  - getFlushableSize
```
public long getFlushableSize()
```
    Description copied from interface: MemStore
    
    On flush, how much memory we will clear. Flush will first clear out the data in snapshot if any (It will take a second flush invocation to clear the current Cell set). If snapshot is empty, current Cell set will be flushed.
    
    Specified by:
    
    getFlushableSize in interface MemStore
    
    Returns:
    size of data that is going to be flushed
  - getSnapshotSize
```
public long getSnapshotSize()
```
    Description copied from interface: MemStore
    
    Return the size of the snapshot(s) if any
    
    Specified by:
    
    getSnapshotSize in interface MemStore
    
    Returns:
    size of the memstore snapshot
  - add
```
public long add(Cell cell)
```
    Write an update
    
    Specified by:
    
    add in interface MemStore
    
    Parameters:
    cell -
    
    Returns:
    approximate size of the passed cell.
  - timeOfOldestEdit
```
public long timeOfOldestEdit()
```
    Specified by:
    
    timeOfOldestEdit in interface MemStore
    
    Returns:
    Oldest timestamp of all the Cells in the MemStore
  - addToCellSet
```
private boolean addToCellSet(Cell e)
```
  - removeFromCellSet
```
private boolean removeFromCellSet(Cell e)
```
  - setOldestEditTimeToNow
```
void setOldestEditTimeToNow()
```
  - internalAdd
```
private long internalAdd(Cell toAdd,
               boolean mslabUsed)
```
    Internal version of add() that doesn't clone Cells with the allocator, and doesn't take the lock. Callers should ensure they already have the read lock taken
    
    Parameters:
    toAdd - the cell to add
    mslabUsed - whether using MSLAB
    
    Returns:
    the heap size change in bytes
  - getCellLength
```
int getCellLength(Cell cell)
```
    Get cell length after serialized in KeyValue
  - maybeCloneWithAllocator
```
private Cell maybeCloneWithAllocator(Cell cell)
```
  - rollback
```
public void rollback(Cell cell)
```
    Remove n key from the memstore. Only cells that have the same key and the same memstoreTS are removed. It is ok to not update timeRangeTracker in this call. It is possible that we can optimize this method by using tailMap/iterator, but since this method is called rarely (only for error recovery), we can leave those optimization for the future.
    
    Specified by:
    
    rollback in interface MemStore
    
    Parameters:
    cell -
  - delete
```
public long delete(Cell deleteCell)
```
    Write a delete
    
    Specified by:
    
    delete in interface MemStore
    
    Parameters:
    deleteCell -
    
    Returns:
    approximate size of the passed key and value.
  - getNextRow
```
Cell getNextRow(Cell cell)
```
    Parameters:
    cell - Find the row that comes after this one. If null, we return the first.
    
    Returns:
    Next row or null if none found.
  - getLowest
```
private Cell getLowest(Cell a,
             Cell b)
```
  - getNextRow
```
private Cell getNextRow(Cell key,
              NavigableSet<Cell> set)
```
  - getRowKeyAtOrBefore
```
public void getRowKeyAtOrBefore(GetClosestRowBeforeTracker state)
```
    Description copied from interface: MemStore
    
    Find the key that matches row exactly, or the one that immediately precedes it. The target row key is set in state.
    
    Specified by:
    
    getRowKeyAtOrBefore in interface MemStore
    
    Parameters:
    state - column/delete tracking state
  - getRowKeyAtOrBefore
```
private void getRowKeyAtOrBefore(NavigableSet<Cell> set,
                       GetClosestRowBeforeTracker state)
```
  - walkForwardInSingleRow
```
private boolean walkForwardInSingleRow(SortedSet<Cell> set,
                             Cell firstOnRow,
                             GetClosestRowBeforeTracker state)
```
  - getRowKeyBefore
```
private void getRowKeyBefore(NavigableSet<Cell> set,
                   GetClosestRowBeforeTracker state)
```
  - updateColumnValue
```
public long updateColumnValue(byte[] row,
                     byte[] family,
                     byte[] qualifier,
                     long newValue,
                     long now)
```
    Only used by tests. TODO: Remove Given the specs of a column, update it, first by inserting a new record, then removing the old one. Since there is only 1 KeyValue involved, the memstoreTS will be set to 0, thus ensuring that they instantly appear to anyone. The underlying store will ensure that the insert/delete each are atomic. A scanner/reader will either get the new value, or the old value and all readers will eventually only see the new value after the old was removed.
    
    Specified by:
    
    updateColumnValue in interface MemStore
    
    Parameters:
    row -
    family -
    qualifier -
    newValue -
    now -
    
    Returns:
    Timestamp
  - upsert
```
public long upsert(Iterable<Cell> cells,
          long readpoint)
```
    Update or insert the specified KeyValues.
    For each KeyValue, insert into MemStore. This will atomically upsert the value for that row/family/qualifier. If a KeyValue did already exist, it will then be removed.
    Currently the memstoreTS is kept at 0 so as each insert happens, it will be immediately visible. May want to change this so it is atomic across all KeyValues.
    This is called under row lock, so Get operations will still see updates atomically. Scans will only see each KeyValue update as atomic.
    
    Specified by:
    
    upsert in interface MemStore
    
    Parameters:
    cells -
    readpoint - readpoint below which we can safely remove duplicate KVs
    
    Returns:
    change in memstore size
  - upsert
```
private long upsert(Cell cell,
          long readpoint)
```
    Inserts the specified KeyValue into MemStore and deletes any existing versions of the same row/family/qualifier as the specified KeyValue.
    First, the specified KeyValue is inserted into the Memstore.
    If there are any existing KeyValues in this MemStore with the same row, family, and qualifier, they are removed.
    Callers must hold the read lock.
    
    Parameters:
    cell -
    
    Returns:
    change in size of MemStore
  - memberOfPreviousRow
```
private DefaultMemStore.Member memberOfPreviousRow(NavigableSet<Cell> set,
                                         GetClosestRowBeforeTracker state,
                                         Cell firstOnRow)
```
  - getScanners
```
public List<KeyValueScanner> getScanners(long readPt)
```
    Specified by:
    
    getScanners in interface MemStore
    
    Returns:
    scanner on memstore and snapshot in this order.
  - shouldSeek
```
public boolean shouldSeek(Scan scan,
                 Store store,
                 long oldestUnexpiredTS)
```
    Check if this memstore may contain the required keys
    
    Parameters:
    scan - scan
    store - holds reference to cf
    oldestUnexpiredTS -
    
    Returns:
    False if the key definitely does not exist in this Memstore
  - heapSizeChange
```
static long heapSizeChange(Cell cell,
                  boolean notpresent)
```
  - keySize
```
private long keySize()
```
  - heapSize
```
public long heapSize()
```
    Get the entire heap usage for this MemStore not including keys in the snapshot.
    
    Specified by:
    
    heapSize in interface HeapSize
    
    Returns:
    Approximate 'exclusive deep size' of implementing object. Includes count of payload and hosting object sizings.
  - size
```
public long size()
```
    Specified by:
    
    size in interface MemStore
    
    Returns:
    Total memory occupied by this MemStore.
  - main
```
public static void main(String[] args)
```
    Code to help figure if our approximation of object heap sizes is close enough. See hbase-900. Fills memstores then waits so user can heap dump and bring up resultant hprof in something like jprofiler which allows you get 'deep size' on objects.
    
    Parameters:
    args - main args

Class DefaultMemStore

Nested Class Summary

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

LOG

USEMSLAB_KEY

USEMSLAB_DEFAULT

MSLAB_CLASS_NAME

conf

cellSet

snapshot

comparator

size

snapshotSize

timeOfOldestEdit

timeRangeTracker

snapshotTimeRangeTracker

allocator

snapshotAllocator

snapshotId

tagsPresent

FIXED_OVERHEAD

DEEP_OVERHEAD

Constructor Detail

DefaultMemStore

DefaultMemStore

Method Detail

dump

snapshot

clearSnapshot

getFlushableSize

getSnapshotSize

add

timeOfOldestEdit

addToCellSet

removeFromCellSet

setOldestEditTimeToNow

internalAdd

getCellLength

maybeCloneWithAllocator

rollback

delete

getNextRow

getLowest

getNextRow

getRowKeyAtOrBefore

getRowKeyAtOrBefore

walkForwardInSingleRow

getRowKeyBefore

updateColumnValue

upsert

upsert

memberOfPreviousRow

getScanners

shouldSeek

heapSizeChange

keySize

heapSize

size

main