@InterfaceAudience.Private public class MemStoreLABImpl extends Object implements MemStoreLAB
The MemStoreLAB is basically a bump-the-pointer allocator that allocates big (2MB) byte[] chunks from and then doles it out to threads that request slices into the array.
The purpose of this class is to combat heap fragmentation in the regionserver. By ensuring that all Cells in a given memstore refer only to large chunks of contiguous memory, we ensure that large blocks get freed up when the memstore is flushed.
Without the MSLAB, the byte array allocated during insertion end up interleaved throughout the heap, and the old generation gets progressively more fragmented until a stop-the-world compacting collection occurs.
TODO: we should probably benchmark whether word-aligning the allocations
would provide a performance improvement - probably would speed up the
Bytes.toLong/Bytes.toInt calls in KeyValue, but some of those are cached
anyway.
The chunks created by this MemStoreLAB can get pooled at ChunkCreator.
When the Chunk comes from pool, it can be either an on heap or an off heap backed chunk. The chunks,
which this MemStoreLAB creates on its own (when no chunk available from pool), those will be
always on heap backed.
| Modifier and Type | Field and Description |
|---|---|
private ChunkCreator |
chunkCreator |
(package private) Set<Integer> |
chunks |
private boolean |
closed |
private AtomicReference<Chunk> |
currChunk |
private int |
dataChunkSize |
private CompactingMemStore.IndexType |
idxType |
private ReentrantLock |
lock |
(package private) static org.slf4j.Logger |
LOG |
private int |
maxAlloc |
private AtomicInteger |
openScannerCount |
private AtomicBoolean |
reclaimed |
CHUNK_POOL_INITIALSIZE_KEY, CHUNK_POOL_MAXSIZE_KEY, CHUNK_SIZE_DEFAULT, CHUNK_SIZE_KEY, INDEX_CHUNK_PERCENTAGE_DEFAULT, INDEX_CHUNK_PERCENTAGE_KEY, MAX_ALLOC_DEFAULT, MAX_ALLOC_KEY, MSLAB_CLASS_NAME, POOL_INITIAL_SIZE_DEFAULT, POOL_MAX_SIZE_DEFAULT, USEMSLAB_DEFAULT, USEMSLAB_KEY| Constructor and Description |
|---|
MemStoreLABImpl() |
MemStoreLABImpl(org.apache.hadoop.conf.Configuration conf) |
| Modifier and Type | Method and Description |
|---|---|
void |
close()
Close this instance since it won't be used any more, try to put the chunks
back to pool
|
private Cell |
copyBBECellInto(ByteBufferExtendedCell cell,
int maxAlloc)
Mostly a duplicate of
copyCellInto(Cell, int)} done for perf sake. |
private static Cell |
copyBBECToChunkCell(ByteBufferExtendedCell cell,
ByteBuffer buf,
int offset,
int len)
Clone the passed cell by copying its data into the passed buf and create a cell with a chunkid
out of it
|
Cell |
copyCellInto(Cell cell)
Allocates slice in this LAB and copy the passed Cell into this area.
|
private Cell |
copyCellInto(Cell cell,
int maxAlloc) |
private static Cell |
copyToChunkCell(Cell cell,
ByteBuffer buf,
int offset,
int len)
Clone the passed cell by copying its data into the passed buf and create a cell with a chunkid
out of it
|
private static Cell |
createChunkCell(ByteBuffer buf,
int offset,
int len,
int tagsLen,
long sequenceId) |
void |
decScannerCount()
Called when closing a scanner on the data of this MemStoreLAB
|
Cell |
forceCopyOfBigCellInto(Cell cell)
When a cell's size is too big (bigger than maxAlloc),
copyCellInto does not allocate it on MSLAB.
|
(package private) Chunk |
getCurrentChunk() |
Chunk |
getNewExternalChunk(ChunkCreator.ChunkType chunkType) |
Chunk |
getNewExternalChunk(int size) |
(package private) Integer |
getNumOfChunksReturnedToPool() |
private Chunk |
getOrMakeChunk()
Get the current chunk, or, if there is no current chunk,
allocate a new one from the JVM.
|
(package private) BlockingQueue<Chunk> |
getPooledChunks() |
void |
incScannerCount()
Called when opening a scanner on the data of this MemStoreLAB
|
boolean |
isOffHeap() |
boolean |
isOnHeap() |
private void |
recycleChunks() |
private void |
tryRetireChunk(Chunk c)
Try to retire the current chunk if it is still
c. |
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, waitisEnabled, newInstancestatic final org.slf4j.Logger LOG
private AtomicReference<Chunk> currChunk
private ReentrantLock lock
private final int dataChunkSize
private final int maxAlloc
private final ChunkCreator chunkCreator
private final CompactingMemStore.IndexType idxType
private volatile boolean closed
private AtomicBoolean reclaimed
private final AtomicInteger openScannerCount
public MemStoreLABImpl()
public MemStoreLABImpl(org.apache.hadoop.conf.Configuration conf)
public Cell copyCellInto(Cell cell)
MemStoreLABcopyCellInto in interface MemStoreLABpublic Cell forceCopyOfBigCellInto(Cell cell)
forceCopyOfBigCellInto in interface MemStoreLABprivate Cell copyBBECellInto(ByteBufferExtendedCell cell, int maxAlloc)
copyCellInto(Cell, int)} done for perf sake. It presumes
ByteBufferExtendedCell instead of Cell so we deal with a specific type rather than the
super generic Cell. Removes instanceof checks. Shrinkage is enough to make this inline where
before it was too big. Uses less CPU. See HBASE-20875 for evidence.copyCellInto(Cell, int)private Cell copyCellInto(Cell cell, int maxAlloc)
private static Cell copyToChunkCell(Cell cell, ByteBuffer buf, int offset, int len)
private static Cell copyBBECToChunkCell(ByteBufferExtendedCell cell, ByteBuffer buf, int offset, int len)
private static Cell createChunkCell(ByteBuffer buf, int offset, int len, int tagsLen, long sequenceId)
public void close()
close in interface MemStoreLABpublic void incScannerCount()
incScannerCount in interface MemStoreLABpublic void decScannerCount()
decScannerCount in interface MemStoreLABprivate void recycleChunks()
private void tryRetireChunk(Chunk c)
c. Postcondition is that curChunk.get()
!= cc - the chunk to retireprivate Chunk getOrMakeChunk()
public Chunk getNewExternalChunk(ChunkCreator.ChunkType chunkType)
getNewExternalChunk in interface MemStoreLABpublic Chunk getNewExternalChunk(int size)
getNewExternalChunk in interface MemStoreLABpublic boolean isOnHeap()
isOnHeap in interface MemStoreLABpublic boolean isOffHeap()
isOffHeap in interface MemStoreLABChunk getCurrentChunk()
BlockingQueue<Chunk> getPooledChunks()
Integer getNumOfChunksReturnedToPool()
Copyright © 2007–2019 The Apache Software Foundation. All rights reserved.