Class MemStoreLABImpl
- All Implemented Interfaces:
MemStoreLAB
The MemStoreLAB is basically a bump-the-pointer allocator that allocates big (2MB) byte[] chunks from and then doles it out to threads that request slices into the array.
The purpose of this class is to combat heap fragmentation in the regionserver. By ensuring that all Cells in a given memstore refer only to large chunks of contiguous memory, we ensure that large blocks get freed up when the memstore is flushed.
Without the MSLAB, the byte array allocated during insertion end up interleaved throughout the heap, and the old generation gets progressively more fragmented until a stop-the-world compacting collection occurs.
TODO: we should probably benchmark whether word-aligning the allocations would provide a
performance improvement - probably would speed up the Bytes.toLong/Bytes.toInt calls in KeyValue,
but some of those are cached anyway. The chunks created by this MemStoreLAB can get pooled at
ChunkCreator
. When the Chunk comes from pool, it can be either an on heap or an off heap
backed chunk. The chunks, which this MemStoreLAB creates on its own (when no chunk available from
pool), those will be always on heap backed.
NOTE:if user requested to work with MSLABs (whether on- or off-heap), in
CompactingMemStore
ctor, the CompactingMemStore.indexType
could only be
CompactingMemStore.IndexType.CHUNK_MAP
,that is to say the immutable segments using MSLABs are going to use
CellChunkMap
as their index.
-
Field Summary
Modifier and TypeFieldDescriptionprivate final ChunkCreator
private final AtomicBoolean
private AtomicReference<Chunk>
private final int
private ReentrantLock
(package private) static final org.slf4j.Logger
private final int
private final AtomicBoolean
private final RefCnt
Its initial value is 1, so it is one bigger than the current count of open scanners which reading data from this MemStoreLAB.Fields inherited from interface org.apache.hadoop.hbase.regionserver.MemStoreLAB
CHUNK_POOL_INITIALSIZE_KEY, CHUNK_POOL_MAXSIZE_KEY, CHUNK_SIZE_DEFAULT, CHUNK_SIZE_KEY, INDEX_CHUNK_SIZE_PERCENTAGE_DEFAULT, INDEX_CHUNK_SIZE_PERCENTAGE_KEY, MAX_ALLOC_DEFAULT, MAX_ALLOC_KEY, MSLAB_CLASS_NAME, POOL_INITIAL_SIZE_DEFAULT, POOL_MAX_SIZE_DEFAULT, USEMSLAB_DEFAULT, USEMSLAB_KEY
-
Constructor Summary
-
Method Summary
Modifier and TypeMethodDescriptionvoid
close()
Close this instance since it won't be used any more, try to put the chunks back to poolprivate ExtendedCell
copyBBECellInto
(ByteBufferExtendedCell cell, int maxAlloc) Mostly a duplicate of#copyCellInto(Cell, int)
} done for perf sake.private static ExtendedCell
copyBBECToChunkCell
(ByteBufferExtendedCell cell, ByteBuffer buf, int offset, int len) Clone the passed cell by copying its data into the passed buf and create a cell with a chunkid out of itcopyCellInto
(ExtendedCell cell) Allocates slice in this LAB and copy the passed Cell into this area.private ExtendedCell
copyCellInto
(ExtendedCell cell, int maxAlloc) private static ExtendedCell
copyToChunkCell
(ExtendedCell cell, ByteBuffer buf, int offset, int len) Clone the passed cell by copying its data into the passed buf and create a cell with a chunkid out of itprivate static ExtendedCell
createChunkCell
(ByteBuffer buf, int offset, int len, int tagsLen, long sequenceId) void
Called when closing a scanner on the data of this MemStoreLABWhen a cell's size is too big (bigger than maxAlloc), copyCellInto does not allocate it on MSLAB.(package private) Chunk
getNewExternalChunk
(int size) getNewExternalChunk
(ChunkCreator.ChunkType chunkType) (package private) Integer
getNumOfChunksReturnedToPool
(Set<Integer> chunksId) private Chunk
Get the current chunk, or, if there is no current chunk, allocate a new one from the JVM.(package private) BlockingQueue<Chunk>
(package private) int
void
Called when opening a scanner on the data of this MemStoreLAB(package private) boolean
isClosed()
boolean
boolean
isOnHeap()
(package private) boolean
private void
private void
Try to retire the current chunk if it is stillc
.
-
Field Details
-
LOG
-
currChunk
-
lock
-
chunks
-
dataChunkSize
-
maxAlloc
-
chunkCreator
-
closed
-
reclaimed
-
refCnt
Its initial value is 1, so it is one bigger than the current count of open scanners which reading data from this MemStoreLAB.
-
-
Constructor Details
-
MemStoreLABImpl
public MemStoreLABImpl() -
MemStoreLABImpl
-
-
Method Details
-
copyCellInto
Description copied from interface:MemStoreLAB
Allocates slice in this LAB and copy the passed Cell into this area. Returns new Cell instance over the copied the data. When this MemStoreLAB can not copy this Cell, it returns null.- Specified by:
copyCellInto
in interfaceMemStoreLAB
-
forceCopyOfBigCellInto
When a cell's size is too big (bigger than maxAlloc), copyCellInto does not allocate it on MSLAB. Since the process of flattening to CellChunkMap assumes that all cells are allocated on MSLAB, during this process, the big cells are copied into MSLAB using this method.- Specified by:
forceCopyOfBigCellInto
in interfaceMemStoreLAB
-
copyBBECellInto
Mostly a duplicate of#copyCellInto(Cell, int)
} done for perf sake. It presumes ByteBufferExtendedCell instead of Cell so we deal with a specific type rather than the super generic Cell. Removes instanceof checks. Shrinkage is enough to make this inline where before it was too big. Uses less CPU. See HBASE-20875 for evidence.- See Also:
-
#copyCellInto(Cell, int)
-
copyCellInto
-
copyToChunkCell
Clone the passed cell by copying its data into the passed buf and create a cell with a chunkid out of it -
copyBBECToChunkCell
private static ExtendedCell copyBBECToChunkCell(ByteBufferExtendedCell cell, ByteBuffer buf, int offset, int len) Clone the passed cell by copying its data into the passed buf and create a cell with a chunkid out of it- See Also:
-
#copyToChunkCell(Cell, ByteBuffer, int, int)
-
createChunkCell
private static ExtendedCell createChunkCell(ByteBuffer buf, int offset, int len, int tagsLen, long sequenceId) -
close
Close this instance since it won't be used any more, try to put the chunks back to pool- Specified by:
close
in interfaceMemStoreLAB
-
getRefCntValue
int getRefCntValue() -
incScannerCount
Called when opening a scanner on the data of this MemStoreLAB- Specified by:
incScannerCount
in interfaceMemStoreLAB
-
decScannerCount
Called when closing a scanner on the data of this MemStoreLAB- Specified by:
decScannerCount
in interfaceMemStoreLAB
-
recycleChunks
-
tryRetireChunk
Try to retire the current chunk if it is stillc
. Postcondition is that curChunk.get() != c- Parameters:
c
- the chunk to retire
-
getOrMakeChunk
Get the current chunk, or, if there is no current chunk, allocate a new one from the JVM. -
getNewExternalChunk
- Specified by:
getNewExternalChunk
in interfaceMemStoreLAB
-
getNewExternalChunk
- Specified by:
getNewExternalChunk
in interfaceMemStoreLAB
-
isOnHeap
- Specified by:
isOnHeap
in interfaceMemStoreLAB
-
isOffHeap
- Specified by:
isOffHeap
in interfaceMemStoreLAB
-
getCurrentChunk
-
getPooledChunks
-
getNumOfChunksReturnedToPool
-
isReclaimed
boolean isReclaimed() -
isClosed
boolean isClosed()
-