Class BucketCache
- All Implemented Interfaces:
Iterable<CachedBlock>
,HeapSize
,BlockCache
BucketAllocator
to allocate/free blocks, and uses BucketCache#ramCache
and BucketCache#backingMap in order to determine if a given element is in the cache. The bucket
cache can use off-heap memory ByteBufferIOEngine
or mmap
ExclusiveMemoryMmapIOEngine
or pmem SharedMemoryMmapIOEngine
or local files
FileIOEngine
to store/read the block data.
Eviction is via a similar algorithm as used in
LruBlockCache
BucketCache can be used as mainly a block cache (see
CombinedBlockCache
), combined with a BlockCache to
decrease CMS GC and heap fragmentation.
It also can be used as a secondary cache (e.g. using a file on ssd/fusionio to store blocks) to enlarge cache space via a victim cache.
-
Nested Class Summary
Modifier and TypeClassDescriptionprivate class
Used to group bucket entries into priority buckets.(package private) static class
Wrapped the delegate ConcurrentMap with maintaining its block's reference count.(package private) static class
Block Entry stored in the memory with key,data and so onprivate static class
(package private) class
-
Field Summary
Modifier and TypeFieldDescription(package private) static final String
private float
Acceptable size of cache (no evictions if size < acceptable)private final AtomicLong
Cache access count (sequential ID)private String
UseMessageDigest
class's encryption algorithms to check persistent file integrity, default algorithm is MD5private static final int
private long
(package private) Map<BlockCacheKey,
BucketEntry> private AtomicBoolean
private final LongAdder
Current number of cached elements(package private) NavigableSet<BlockCacheKey>
private final long
Approximate block sizeprivate BucketAllocator
private long
private final long
private boolean
Flag if the cache is enabled or not...private BucketCachePersister
private final BucketCacheStats
private static final float
private static final int
static final int
private static final float
private static final String
private static final int
(package private) static final float
(package private) static final float
(package private) static final float
private static final long
(package private) static final float
Priority buckets(package private) static final int
(package private) static final int
(package private) static final String
private float
Free this floating point factor of extra blocks when evicting.private static final String
private boolean
Volatile boolean to track if free space is in process or notprivate final Lock
Map of hFile -> Region -> File size.private final LongAdder
(package private) final IOEngine
private long
private final int
Duration of IO errors tolerated before we disable cache, 1 min as default(package private) static AtomicBoolean
private static final org.slf4j.Logger
(package private) static final String
private float
In-memory bucket size(package private) static final String
private float
Minimum threshold of cache (when evicting, evict until size < min)(package private) static final String
private float
Multiple access bucket size(package private) final IdReadWriteLock<Long>
A ReentrantReadWriteLock to lock on a particular block identified by offset.private final String
private static final String
private long
(package private) final BucketCache.RAMCache
private final LongAdder
Map of region -> total size of the region prefetched on this region server.private final ScheduledExecutorService
Statistics thread schedule pool (for heavy debugging, could remove)(package private) static final String
Priority buckets configprivate float
Single access bucket sizeprivate static final int
Statistics thread(package private) final ArrayList<BlockingQueue<BucketCache.RAMQueueEntry>>
A list of writer queues.(package private) final BucketCache.WriterThread[]
-
Constructor Summary
ConstructorDescriptionBucketCache
(String ioEngineName, long capacity, int blockSize, int[] bucketSizes, int writerThreadNum, int writerQLen, String persistencePath) BucketCache
(String ioEngineName, long capacity, int blockSize, int[] bucketSizes, int writerThreadNum, int writerQLen, String persistencePath, int ioErrorsTolerationDuration, org.apache.hadoop.conf.Configuration conf) -
Method Summary
Modifier and TypeMethodDescriptionlong
(package private) void
blockEvicted
(BlockCacheKey cacheKey, BucketEntry bucketEntry, boolean decrementBlockNumber, boolean evictedByEvictionProcess) This method is invoked after the bucketEntry is removed frombackingMap
blockFitsIntoTheCache
(HFileBlock block) Checks whether there's enough space left in the cache to accommodate the passed block.private int
bucketSizesAboveThresholdCount
(float minFactor) Return the count of bucketSizeinfos still need free spacevoid
cacheBlock
(BlockCacheKey cacheKey, Cacheable buf) Cache the block with the specified name and buffer.void
cacheBlock
(BlockCacheKey cacheKey, Cacheable cachedItem, boolean inMemory) Cache the block with the specified name and buffer.void
cacheBlock
(BlockCacheKey cacheKey, Cacheable cachedItem, boolean inMemory, boolean waitWhenCache) Cache the block with the specified name and buffer.void
cacheBlockWithWait
(BlockCacheKey cacheKey, Cacheable cachedItem, boolean inMemory, boolean wait) Cache the block to ramCacheprotected void
cacheBlockWithWaitInternal
(BlockCacheKey cacheKey, Cacheable cachedItem, boolean inMemory, boolean wait) private void
Check whether we tolerate IO error this time.private ByteBuffAllocator.Recycler
createRecycler
(BucketEntry bucketEntry) Create theByteBuffAllocator.Recycler
forBucketEntry.refCnt
,which would be used asRefCnt.recycler
ofHFileBlock#buf
returned fromgetBlock(org.apache.hadoop.hbase.io.hfile.BlockCacheKey, boolean, boolean, boolean)
.private FileInputStream
deleteFileOnClose
(File file) Create an input stream that deletes the file after reading it.private void
Used to shut down the cache -or- turn it off in the case of something broken.(package private) void
doDrain
(List<BucketCache.RAMQueueEntry> entries, ByteBuffer metaBuff) Flush the entries in ramCache to IOEngine and add bucket entry to backingMap.private boolean
doEvictBlock
(BlockCacheKey cacheKey, BucketEntry bucketEntry, boolean evictedByEvictionProcess) private void
boolean
evictBlock
(BlockCacheKey cacheKey) Try to evict the block fromBlockCache
by force.boolean
evictBlockIfNoRpcReferenced
(BlockCacheKey blockCacheKey) NOTE: This method is only for test.int
evictBlocksByHfileName
(String hfileName) Evicts all blocks for a specific HFile.(package private) boolean
evictBucketEntryIfNoRpcReferenced
(BlockCacheKey blockCacheKey, BucketEntry bucketEntry) void
fileCacheCompleted
(org.apache.hadoop.fs.Path filePath, long size) protected void
finalize()
Needed mostly for UTs that might run in the same VM and create different BucketCache instances on different UT methods.(package private) void
freeBucketEntry
(BucketEntry bucketEntry) Free the {BucketEntry
actually,which could only be invoked when theBucketEntry.refCnt
becoming 0.private void
freeEntireBuckets
(int completelyFreeBucketsNeeded) This method will find the buckets that are minimally occupied and are not reference counted and will free them completely without any constraint on the access times of the elements, and as a process will completely free at most the number of buckets passed, sometimes it might not due to changing refCounts(package private) void
Free the space if the used size reaches acceptableSize() or one size block couldn't be allocated.(package private) float
protected String
private Set<BlockCacheKey>
getAllCacheKeysForFile
(String hfileName) private static String
Prepare and return a warning message for Bucket Allocator ExceptiongetBlock
(BlockCacheKey key, boolean caching, boolean repeat, boolean updateCacheMetrics) Get the buffer of the block with the specified key.Returns The list of sub blockcaches that make up this one; returns null if no sub caches.long
Returns the number of blocks currently cached in the block cache.Returns an Optional containing the size of the block related to the passed key.static Optional<BucketCache>
getBucketCacheFromCacheConfig
(CacheConfig cacheConf) long
Returns the occupied size of data blocks, in bytes.long
Returns the occupied size of the block cache, in bytes.long
Returns the number of data blocks currently cached in the block cache.(package private) float
long
Returns the free size of the block cache, in bytes.Returns an Optional containing the map of files that have been fully cached (all its blocks are present in the cache.private IOEngine
getIOEngineFromName
(String ioEngineName, long capacity, String persistencePath) Get the IOEngine from the IO engine namelong
Returns the Max size of the block cache, in bytes.(package private) float
(package private) float
(package private) float
(package private) long
getPartitionSize
(float partitionFactor) (package private) static List<BucketCache.RAMQueueEntry>
getRAMQueueEntries
(BlockingQueue<BucketCache.RAMQueueEntry> q, List<BucketCache.RAMQueueEntry> receptacle) Blocks until elements available inq
then tries to grab as many as possible before returning.long
int
getRpcRefCount
(BlockCacheKey cacheKey) (package private) float
getStats()
Get the statistics for this block cache.long
heapSize()
Return the approximate 'exclusive deep size' of implementing object.Checks whether the block for the passed key is already cached.(package private) boolean
boolean
boolean
boolean
iterator()
Returns Iterator over the blocks in the cache.private void
join()
void
logStats()
void
notifyFileBlockEvicted
(String fileName) Notifies the cache implementation that the given file had a block evictedvoid
notifyFileCachingCompleted
(org.apache.hadoop.fs.Path fileName, int totalBlockCount, int dataBlockCount, long size) Notifies the cache implementation that the given file has been fully cached (all its blocks made into the cache).private void
parsePB
(org.apache.hadoop.hbase.shaded.protobuf.generated.BucketCacheProtos.BucketCacheEntry proto) (package private) void
protected void
putIntoBackingMap
(BlockCacheKey key, BucketEntry bucketEntry) Put the new bucket entry into backingMap.private void
removeFileFromPrefetch
(String hfileName) protected boolean
removeFromRamCache
(BlockCacheKey cacheKey) private void
retrieveFromFile
(int[] bucketSizes) private void
void
setCacheInconsistent
(boolean setCacheInconsistent) shouldCacheFile
(String fileName) Checks whether blocks for the passed file should be cached or not.protected boolean
shouldReplaceExistingCacheBlock
(BlockCacheKey cacheKey, Cacheable newBlock) void
shutdown()
Shutdown the cache.long
size()
Returns the total size of the block cache, in bytes.(package private) void
protected void
Called by the constructor to start the writer threads.(package private) void
Only used in testprivate void
private void
verifyCapacityAndClasses
(long capacitySize, String ioclass, String mapclass) Methods inherited from class java.lang.Object
clone, equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
Methods inherited from interface org.apache.hadoop.hbase.io.hfile.BlockCache
isMetaBlock
Methods inherited from interface java.lang.Iterable
forEach, spliterator
-
Field Details
-
LOG
-
SINGLE_FACTOR_CONFIG_NAME
Priority buckets config- See Also:
-
MULTI_FACTOR_CONFIG_NAME
- See Also:
-
MEMORY_FACTOR_CONFIG_NAME
- See Also:
-
EXTRA_FREE_FACTOR_CONFIG_NAME
- See Also:
-
ACCEPT_FACTOR_CONFIG_NAME
- See Also:
-
MIN_FACTOR_CONFIG_NAME
- See Also:
-
DEFAULT_SINGLE_FACTOR
Priority buckets- See Also:
-
DEFAULT_MULTI_FACTOR
- See Also:
-
DEFAULT_MEMORY_FACTOR
- See Also:
-
DEFAULT_MIN_FACTOR
- See Also:
-
DEFAULT_EXTRA_FREE_FACTOR
- See Also:
-
DEFAULT_ACCEPT_FACTOR
- See Also:
-
DEFAULT_FREE_ENTIRE_BLOCK_FACTOR
- See Also:
-
statThreadPeriod
Statistics thread- See Also:
-
DEFAULT_WRITER_THREADS
- See Also:
-
DEFAULT_WRITER_QUEUE_ITEMS
- See Also:
-
ioEngine
-
ramCache
-
backingMap
-
backingMapValidated
-
fullyCachedFiles
Map of hFile -> Region -> File size. This map is used to track all files completed prefetch, together with the region those belong to and the total cached size for the region.TestBlockEvictionOnRegionMovement -
regionCachedSizeMap
Map of region -> total size of the region prefetched on this region server. This is the total size of hFiles for this region prefetched on this region server -
cachePersister
-
cacheEnabled
Flag if the cache is enabled or not... We shut it off if there are IO errors for some time, so that Bucket IO exceptions/errors don't bring down the HBase server. -
writerQueues
A list of writer queues. We have a queue perBucketCache.WriterThread
we have running. In other words, the work adding blocks to the BucketCache is divided up amongst the running WriterThreads. Its done by taking hash of the cache key modulo queue count. WriterThread when it runs takes whatever has been recently added and 'drains' the entries to the BucketCache. It then updates the ramCache and backingMap accordingly. -
writerThreads
-
freeInProgress
Volatile boolean to track if free space is in process or not -
freeSpaceLock
-
realCacheSize
-
heapSize
-
blockNumber
Current number of cached elements -
accessCount
Cache access count (sequential ID) -
DEFAULT_CACHE_WAIT_TIME
- See Also:
-
cacheStats
-
persistencePath
-
isCacheInconsistent
-
cacheCapacity
-
blockSize
Approximate block size -
ioErrorsTolerationDuration
Duration of IO errors tolerated before we disable cache, 1 min as default -
DEFAULT_ERROR_TOLERATION_DURATION
- See Also:
-
ioErrorStartTime
-
offsetLock
A ReentrantReadWriteLock to lock on a particular block identified by offset. The purpose of this is to avoid freeing the block which is being read.Key set of offsets in BucketCache is limited so soft reference is the best choice here.
-
blocksByHFile
-
scheduleThreadPool
Statistics thread schedule pool (for heavy debugging, could remove) -
bucketAllocator
-
acceptableFactor
Acceptable size of cache (no evictions if size < acceptable) -
minFactor
Minimum threshold of cache (when evicting, evict until size < min) -
extraFreeFactor
Free this floating point factor of extra blocks when evicting. For example free the number of blocks requested * (1 + extraFreeFactor) -
singleFactor
Single access bucket size -
multiFactor
Multiple access bucket size -
memoryFactor
In-memory bucket size -
bucketcachePersistInterval
-
FILE_VERIFY_ALGORITHM
- See Also:
-
DEFAULT_FILE_VERIFY_ALGORITHM
- See Also:
-
QUEUE_ADDITION_WAIT_TIME
- See Also:
-
DEFAULT_QUEUE_ADDITION_WAIT_TIME
- See Also:
-
queueAdditionWaitTime
-
algorithm
UseMessageDigest
class's encryption algorithms to check persistent file integrity, default algorithm is MD5 -
allocFailLogPrevTs
-
ALLOCATION_FAIL_LOG_TIME_PERIOD
- See Also:
-
-
Constructor Details
-
BucketCache
public BucketCache(String ioEngineName, long capacity, int blockSize, int[] bucketSizes, int writerThreadNum, int writerQLen, String persistencePath) throws IOException - Throws:
IOException
-
BucketCache
public BucketCache(String ioEngineName, long capacity, int blockSize, int[] bucketSizes, int writerThreadNum, int writerQLen, String persistencePath, int ioErrorsTolerationDuration, org.apache.hadoop.conf.Configuration conf) throws IOException - Throws:
IOException
-
-
Method Details
-
sanityCheckConfigs
-
startWriterThreads
Called by the constructor to start the writer threads. Used by tests that need to override starting the threads. -
startBucketCachePersisterThread
-
isCacheEnabled
boolean isCacheEnabled() -
getMaxSize
Description copied from interface:BlockCache
Returns the Max size of the block cache, in bytes.- Specified by:
getMaxSize
in interfaceBlockCache
- Returns:
- size of cache, in bytes
-
getIoEngine
-
getIOEngineFromName
private IOEngine getIOEngineFromName(String ioEngineName, long capacity, String persistencePath) throws IOException Get the IOEngine from the IO engine name- Returns:
- the IOEngine
- Throws:
IOException
-
isCachePersistenceEnabled
-
cacheBlock
Cache the block with the specified name and buffer.- Specified by:
cacheBlock
in interfaceBlockCache
- Parameters:
cacheKey
- block's cache keybuf
- block buffer
-
cacheBlock
Cache the block with the specified name and buffer.- Specified by:
cacheBlock
in interfaceBlockCache
- Parameters:
cacheKey
- block's cache keycachedItem
- block bufferinMemory
- if block is in-memory
-
cacheBlock
public void cacheBlock(BlockCacheKey cacheKey, Cacheable cachedItem, boolean inMemory, boolean waitWhenCache) Cache the block with the specified name and buffer.- Specified by:
cacheBlock
in interfaceBlockCache
- Parameters:
cacheKey
- block's cache keycachedItem
- block bufferinMemory
- if block is in-memorywaitWhenCache
- Whether to wait for the cache to be flushed mainly when BucketCache is configured.
-
cacheBlockWithWait
public void cacheBlockWithWait(BlockCacheKey cacheKey, Cacheable cachedItem, boolean inMemory, boolean wait) Cache the block to ramCache- Parameters:
cacheKey
- block's cache keycachedItem
- block bufferinMemory
- if block is in-memorywait
- if true, blocking wait when queue is full
-
shouldReplaceExistingCacheBlock
-
cacheBlockWithWaitInternal
protected void cacheBlockWithWaitInternal(BlockCacheKey cacheKey, Cacheable cachedItem, boolean inMemory, boolean wait) -
getBlock
public Cacheable getBlock(BlockCacheKey key, boolean caching, boolean repeat, boolean updateCacheMetrics) Get the buffer of the block with the specified key.- Specified by:
getBlock
in interfaceBlockCache
- Parameters:
key
- block's cache keycaching
- true if the caller caches blocks on cache missesrepeat
- Whether this is a repeat lookup for the same blockupdateCacheMetrics
- Whether we should update cache metrics or not- Returns:
- buffer of specified cache key, or null if not in cache
-
blockEvicted
void blockEvicted(BlockCacheKey cacheKey, BucketEntry bucketEntry, boolean decrementBlockNumber, boolean evictedByEvictionProcess) This method is invoked after the bucketEntry is removed frombackingMap
-
freeBucketEntry
Free the {BucketEntry
actually,which could only be invoked when theBucketEntry.refCnt
becoming 0. -
evictBlock
Try to evict the block fromBlockCache
by force. We'll call this in few cases:
1. Close an HFile, and clear all cached blocks.
2. CallAdmin.clearBlockCache(TableName)
to clear all blocks for a given table.
Firstly, we'll try to remove the block from RAMCache,and then try to evict from backingMap. Here we evict the block from backingMap immediately, but only free the reference from bucket cache by calling
BucketEntry.markedAsEvicted
. If there're still some RPC referring this block, block can only be de-allocated when all of them release the block.NOTICE: we need to grab the write offset lock firstly before releasing the reference from bucket cache. if we don't, we may read an
BucketEntry
with refCnt = 0 whengetBlock(BlockCacheKey, boolean, boolean, boolean)
, it's a memory leak.- Specified by:
evictBlock
in interfaceBlockCache
- Parameters:
cacheKey
- Block to evict- Returns:
- true to indicate whether we've evicted successfully or not.
-
doEvictBlock
private boolean doEvictBlock(BlockCacheKey cacheKey, BucketEntry bucketEntry, boolean evictedByEvictionProcess) Evict theBlockCacheKey
andBucketEntry
frombackingMap
andramCache
.
NOTE:When Evict frombackingMap
,only the matchedBlockCacheKey
andBucketEntry
could be removed.- Parameters:
cacheKey
-BlockCacheKey
to evict.bucketEntry
-BucketEntry
matchedBlockCacheKey
to evict.- Returns:
- true to indicate whether we've evicted successfully or not.
-
createRecycler
Create the
ByteBuffAllocator.Recycler
forBucketEntry.refCnt
,which would be used asRefCnt.recycler
ofHFileBlock#buf
returned fromgetBlock(org.apache.hadoop.hbase.io.hfile.BlockCacheKey, boolean, boolean, boolean)
. NOTE: forgetBlock(org.apache.hadoop.hbase.io.hfile.BlockCacheKey, boolean, boolean, boolean)
,theRefCnt.recycler
ofHFileBlock#buf
frombackingMap
andramCache
are different: 1.ForRefCnt.recycler
ofHFileBlock#buf
frombackingMap
, it is the return value of currentcreateRecycler(org.apache.hadoop.hbase.io.hfile.bucket.BucketEntry)
method. 2.ForRefCnt.recycler
ofHFileBlock#buf
fromramCache
, it isByteBuffAllocator.putbackBuffer(java.nio.ByteBuffer)
. -
evictBlockIfNoRpcReferenced
NOTE: This method is only for test. -
evictBucketEntryIfNoRpcReferenced
EvictBlockCacheKey
and its correspondingBucketEntry
only ifBucketEntry.isRpcRef()
is false.
NOTE:When evict frombackingMap
,only the matchedBlockCacheKey
andBucketEntry
could be removed.- Parameters:
blockCacheKey
-BlockCacheKey
to evict.bucketEntry
-BucketEntry
matchedBlockCacheKey
to evict.- Returns:
- true to indicate whether we've evicted successfully or not.
-
removeFromRamCache
-
isCacheInconsistent
-
setCacheInconsistent
-
logStats
-
getRealCacheSize
-
acceptableSize
-
getPartitionSize
-
bucketSizesAboveThresholdCount
Return the count of bucketSizeinfos still need free space -
freeEntireBuckets
This method will find the buckets that are minimally occupied and are not reference counted and will free them completely without any constraint on the access times of the elements, and as a process will completely free at most the number of buckets passed, sometimes it might not due to changing refCounts- Parameters:
completelyFreeBucketsNeeded
- number of buckets to free
-
freeSpace
Free the space if the used size reaches acceptableSize() or one size block couldn't be allocated. When freeing the space, we use the LRU algorithm and ensure there must be some blocks evicted- Parameters:
why
- Why we are being called
-
putIntoBackingMap
Put the new bucket entry into backingMap. Notice that we are allowed to replace the existing cache with a new block for the same cache key. there's a corner case: one thread cache a block in ramCache, copy to io-engine and add a bucket entry to backingMap. Caching another new block with the same cache key do the same thing for the same cache key, so if not evict the previous bucket entry, then memory leak happen because the previous bucketEntry is gone but the bucketAllocator do not free its memory.- Parameters:
key
- Block cache keybucketEntry
- Bucket entry to put into backingMap.- See Also:
-
getAllocationFailWarningMessage
private static String getAllocationFailWarningMessage(BucketAllocatorException fle, BucketCache.RAMQueueEntry re) Prepare and return a warning message for Bucket Allocator Exception- Parameters:
fle
- The exceptionre
- The RAMQueueEntry for which the exception was thrown.- Returns:
- A warning message created from the input RAMQueueEntry object.
-
doDrain
void doDrain(List<BucketCache.RAMQueueEntry> entries, ByteBuffer metaBuff) throws InterruptedException Flush the entries in ramCache to IOEngine and add bucket entry to backingMap. Process all that are passed in even if failure being sure to remove from ramCache else we'll never undo the references and we'll OOME.- Parameters:
entries
- Presumes list passed in here will be processed by this invocation only. No interference expected.- Throws:
InterruptedException
-
getRAMQueueEntries
static List<BucketCache.RAMQueueEntry> getRAMQueueEntries(BlockingQueue<BucketCache.RAMQueueEntry> q, List<BucketCache.RAMQueueEntry> receptacle) throws InterruptedException Blocks until elements available inq
then tries to grab as many as possible before returning.- Parameters:
receptacle
- Where to stash the elements taken from queue. We clear before we use it just in case.q
- The queue to take from.- Returns:
receptacle
laden with elements taken from the queue or empty if none found.- Throws:
InterruptedException
-
persistToFile
- Throws:
IOException
- See Also:
-
isCachePersistent
-
getRegionCachedInfo
-
retrieveFromFile
- Throws:
IOException
- See Also:
-
updateRegionSizeMapWhileRetrievingFromFile
-
dumpPrefetchList
-
deleteFileOnClose
Create an input stream that deletes the file after reading it. Use in try-with-resources to avoid this pattern where an exception thrown from a finally block may mask earlier exceptions:File f = ... try (FileInputStream fis = new FileInputStream(f)) { // use the input stream } finally { if (!f.delete()) throw new IOException("failed to delete"); }
- Parameters:
file
- the file to read and delete- Returns:
- a FileInputStream for the given file
- Throws:
IOException
- if there is a problem creating the stream
-
verifyCapacityAndClasses
private void verifyCapacityAndClasses(long capacitySize, String ioclass, String mapclass) throws IOException - Throws:
IOException
-
parsePB
private void parsePB(org.apache.hadoop.hbase.shaded.protobuf.generated.BucketCacheProtos.BucketCacheEntry proto) throws IOException - Throws:
IOException
-
checkIOErrorIsTolerated
Check whether we tolerate IO error this time. If the duration of IOEngine throwing errors exceeds ioErrorsDurationTimeTolerated, we will disable the cache -
disableCache
Used to shut down the cache -or- turn it off in the case of something broken. -
join
- Throws:
InterruptedException
-
shutdown
Description copied from interface:BlockCache
Shutdown the cache.- Specified by:
shutdown
in interfaceBlockCache
-
finalize
Needed mostly for UTs that might run in the same VM and create different BucketCache instances on different UT methods. -
getStats
Description copied from interface:BlockCache
Get the statistics for this block cache.- Specified by:
getStats
in interfaceBlockCache
-
getAllocator
-
heapSize
Description copied from interface:HeapSize
Return the approximate 'exclusive deep size' of implementing object. Includes count of payload and hosting object sizings. -
size
Description copied from interface:BlockCache
Returns the total size of the block cache, in bytes.- Specified by:
size
in interfaceBlockCache
- Returns:
- size of cache, in bytes
-
getCurrentDataSize
Description copied from interface:BlockCache
Returns the occupied size of data blocks, in bytes.- Specified by:
getCurrentDataSize
in interfaceBlockCache
- Returns:
- occupied space in cache, in bytes
-
getFreeSize
Description copied from interface:BlockCache
Returns the free size of the block cache, in bytes.- Specified by:
getFreeSize
in interfaceBlockCache
- Returns:
- free space in cache, in bytes
-
getBlockCount
Description copied from interface:BlockCache
Returns the number of blocks currently cached in the block cache.- Specified by:
getBlockCount
in interfaceBlockCache
- Returns:
- number of blocks in the cache
-
getDataBlockCount
Description copied from interface:BlockCache
Returns the number of data blocks currently cached in the block cache.- Specified by:
getDataBlockCount
in interfaceBlockCache
- Returns:
- number of blocks in the cache
-
getCurrentSize
Description copied from interface:BlockCache
Returns the occupied size of the block cache, in bytes.- Specified by:
getCurrentSize
in interfaceBlockCache
- Returns:
- occupied space in cache, in bytes
-
getAlgorithm
-
evictBlocksByHfileName
Evicts all blocks for a specific HFile.This is used for evict-on-close to remove all blocks of a specific HFile.
- Specified by:
evictBlocksByHfileName
in interfaceBlockCache
- Returns:
- the number of blocks evicted
-
getAllCacheKeysForFile
-
stopWriterThreads
Only used in test- Throws:
InterruptedException
-
iterator
Description copied from interface:BlockCache
Returns Iterator over the blocks in the cache.- Specified by:
iterator
in interfaceBlockCache
- Specified by:
iterator
in interfaceIterable<CachedBlock>
-
getBlockCaches
Description copied from interface:BlockCache
Returns The list of sub blockcaches that make up this one; returns null if no sub caches.- Specified by:
getBlockCaches
in interfaceBlockCache
-
getRpcRefCount
-
getAcceptableFactor
float getAcceptableFactor() -
getMinFactor
float getMinFactor() -
getExtraFreeFactor
float getExtraFreeFactor() -
getSingleFactor
float getSingleFactor() -
getMultiFactor
float getMultiFactor() -
getMemoryFactor
float getMemoryFactor() -
getPersistencePath
-
getBackingMap
-
getBackingMapValidated
-
getFullyCachedFiles
Description copied from interface:BlockCache
Returns an Optional containing the map of files that have been fully cached (all its blocks are present in the cache. This method may not be overridden by all implementing classes. In such cases, the returned Optional will be empty.- Specified by:
getFullyCachedFiles
in interfaceBlockCache
- Returns:
- empty optional if this method is not supported, otherwise the returned optional contains a map of all files that have been fully cached.
-
getBucketCacheFromCacheConfig
-
notifyFileCachingCompleted
public void notifyFileCachingCompleted(org.apache.hadoop.fs.Path fileName, int totalBlockCount, int dataBlockCount, long size) Description copied from interface:BlockCache
Notifies the cache implementation that the given file has been fully cached (all its blocks made into the cache).- Specified by:
notifyFileCachingCompleted
in interfaceBlockCache
- Parameters:
fileName
- the file that has been completely cached.
-
notifyFileBlockEvicted
Description copied from interface:BlockCache
Notifies the cache implementation that the given file had a block evicted- Specified by:
notifyFileBlockEvicted
in interfaceBlockCache
- Parameters:
fileName
- the file had a block evicted.
-
blockFitsIntoTheCache
Description copied from interface:BlockCache
Checks whether there's enough space left in the cache to accommodate the passed block. This method may not be overridden by all implementing classes. In such cases, the returned Optional will be empty. For subclasses implementing this logic, the returned Optional would contain the boolean value reflecting if the passed block fits into the remaining cache space available.- Specified by:
blockFitsIntoTheCache
in interfaceBlockCache
- Parameters:
block
- the block we want to check if fits into the cache.- Returns:
- empty optional if this method is not supported, otherwise the returned optional contains the boolean value informing if the block fits into the cache available space.
-
shouldCacheFile
Description copied from interface:BlockCache
Checks whether blocks for the passed file should be cached or not. This method may not be overridden by all implementing classes. In such cases, the returned Optional will be empty. For subclasses implementing this logic, the returned Optional would contain the boolean value reflecting if the passed file should indeed be cached.- Specified by:
shouldCacheFile
in interfaceBlockCache
- Parameters:
fileName
- to check if it should be cached.- Returns:
- empty optional if this method is not supported, otherwise the returned optional contains the boolean value informing if the file should be cached.
-
isAlreadyCached
Description copied from interface:BlockCache
Checks whether the block for the passed key is already cached. This method may not be overridden by all implementing classes. In such cases, the returned Optional will be empty. For subclasses implementing this logic, the returned Optional would contain the boolean value reflecting if the block for the passed key is already cached or not.- Specified by:
isAlreadyCached
in interfaceBlockCache
- Parameters:
key
- for the block we want to check if it's already in the cache.- Returns:
- empty optional if this method is not supported, otherwise the returned optional contains the boolean value informing if the block is already cached.
-
getBlockSize
Description copied from interface:BlockCache
Returns an Optional containing the size of the block related to the passed key. If the block is not in the cache, returned optional will be empty. Also, this method may not be overridden by all implementing classes. In such cases, the returned Optional will be empty.- Specified by:
getBlockSize
in interfaceBlockCache
- Parameters:
key
- for the block we want to check if it's already in the cache.- Returns:
- empty optional if this method is not supported, otherwise the returned optional contains the boolean value informing if the block is already cached.
-
removeFileFromPrefetch
-
fileCacheCompleted
-