BucketCache (Apache HBase 2.5.0 API)

java.lang.Object
- org.apache.hadoop.hbase.io.hfile.bucket.BucketCache

All Implemented Interfaces:

Iterable<CachedBlock>, HeapSize, BlockCache
```
@InterfaceAudience.Private
public class BucketCache
extends Object
implements BlockCache, HeapSize
```
BucketCache uses BucketAllocator to allocate/free blocks, and uses BucketCache#ramCache and BucketCache#backingMap in order to determine if a given element is in the cache. The bucket cache can use off-heap memory ByteBufferIOEngine or mmap ExclusiveMemoryMmapIOEngine or pmem SharedMemoryMmapIOEngine or local files FileIOEngine to store/read the block data.
Eviction is via a similar algorithm as used in LruBlockCache
BucketCache can be used as mainly a block cache (see CombinedBlockCache), combined with a BlockCache to decrease CMS GC and heap fragmentation.
It also can be used as a secondary cache (e.g. using a file on ssd/fusionio to store blocks) to enlarge cache space via a victim cache.

Nested Class Summary

Nested Classes
Modifier and Type	Class and Description
`private class`	`BucketCache.BucketEntryGroup` Used to group bucket entries into priority buckets.
`(package private) static class`	`BucketCache.RAMCache` Wrapped the delegate ConcurrentMap with maintaining its block's reference count.
`(package private) static class`	`BucketCache.RAMQueueEntry` Block Entry stored in the memory with key,data and so on
`private static class`	`BucketCache.StatisticsThread`
`(package private) class`	`BucketCache.WriterThread`

Field Summary

Fields
Modifier and Type	Field and Description
`(package private) static String`	`ACCEPT_FACTOR_CONFIG_NAME`
`private float`	`acceptableFactor` Acceptable size of cache (no evictions if size < acceptable)
`private AtomicLong`	`accessCount` Cache access count (sequential ID)
`private String`	`algorithm` Use `MessageDigest` class's encryption algorithms to check persistent file integrity, default algorithm is MD5
`private static int`	`ALLOCATION_FAIL_LOG_TIME_PERIOD`
`private long`	`allocFailLogPrevTs`
`(package private) ConcurrentHashMap<BlockCacheKey,BucketEntry>`	`backingMap`
`private LongAdder`	`blockNumber` Current number of cached elements
`private NavigableSet<BlockCacheKey>`	`blocksByHFile`
`private long`	`blockSize` Approximate block size
`private BucketAllocator`	`bucketAllocator`
`private long`	`cacheCapacity`
`private boolean`	`cacheEnabled` Flag if the cache is enabled or not...
`private BucketCacheStats`	`cacheStats`
`private static float`	`DEFAULT_ACCEPT_FACTOR`
`private static int`	`DEFAULT_CACHE_WAIT_TIME`
`static int`	`DEFAULT_ERROR_TOLERATION_DURATION`
`private static float`	`DEFAULT_EXTRA_FREE_FACTOR`
`private static String`	`DEFAULT_FILE_VERIFY_ALGORITHM`
`private static int`	`DEFAULT_FREE_ENTIRE_BLOCK_FACTOR`
`(package private) static float`	`DEFAULT_MEMORY_FACTOR`
`(package private) static float`	`DEFAULT_MIN_FACTOR`
`(package private) static float`	`DEFAULT_MULTI_FACTOR`
`(package private) static float`	`DEFAULT_SINGLE_FACTOR` Priority buckets
`(package private) static int`	`DEFAULT_WRITER_QUEUE_ITEMS`
`(package private) static int`	`DEFAULT_WRITER_THREADS`
`(package private) static String`	`EXTRA_FREE_FACTOR_CONFIG_NAME`
`private float`	`extraFreeFactor` Free this floating point factor of extra blocks when evicting.
`private static String`	`FILE_VERIFY_ALGORITHM`
`private boolean`	`freeInProgress` Volatile boolean to track if free space is in process or not
`private Lock`	`freeSpaceLock`
`private LongAdder`	`heapSize`
`(package private) IOEngine`	`ioEngine`
`private long`	`ioErrorStartTime`
`private int`	`ioErrorsTolerationDuration` Duration of IO errors tolerated before we disable cache, 1 min as default
`private static org.slf4j.Logger`	`LOG`
`(package private) static String`	`MEMORY_FACTOR_CONFIG_NAME`
`private float`	`memoryFactor` In-memory bucket size
`(package private) static String`	`MIN_FACTOR_CONFIG_NAME`
`private float`	`minFactor` Minimum threshold of cache (when evicting, evict until size < min)
`(package private) static String`	`MULTI_FACTOR_CONFIG_NAME`
`private float`	`multiFactor` Multiple access bucket size
`(package private) IdReadWriteLock<Long>`	`offsetLock` A ReentrantReadWriteLock to lock on a particular block identified by offset.
`private String`	`persistencePath`
`(package private) BucketCache.RAMCache`	`ramCache`
`private LongAdder`	`realCacheSize`
`private ScheduledExecutorService`	`scheduleThreadPool` Statistics thread schedule pool (for heavy debugging, could remove)
`(package private) static String`	`SINGLE_FACTOR_CONFIG_NAME` Priority buckets config
`private float`	`singleFactor` Single access bucket size
`private static int`	`statThreadPeriod` Statistics thread
`(package private) boolean`	`wait_when_cache` Used in tests.
`(package private) ArrayList<BlockingQueue<BucketCache.RAMQueueEntry>>`	`writerQueues` A list of writer queues.
`(package private) BucketCache.WriterThread[]`	`writerThreads`

Constructor Summary

Constructors
Constructor and Description
`BucketCache(String ioEngineName, long capacity, int blockSize, int[] bucketSizes, int writerThreadNum, int writerQLen, String persistencePath)`
`BucketCache(String ioEngineName, long capacity, int blockSize, int[] bucketSizes, int writerThreadNum, int writerQLen, String persistencePath, int ioErrorsTolerationDuration, org.apache.hadoop.conf.Configuration conf)`

Method Summary

All Methods Static Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`long`	`acceptableSize()`
`(package private) void`	`blockEvicted(BlockCacheKey cacheKey, BucketEntry bucketEntry, boolean decrementBlockNumber, boolean evictedByEvictionProcess)` This method is invoked after the bucketEntry is removed from `backingMap`
`private int`	`bucketSizesAboveThresholdCount(float minFactor)` Return the count of bucketSizeinfos still need free space
`void`	`cacheBlock(BlockCacheKey cacheKey, Cacheable buf)` Cache the block with the specified name and buffer.
`void`	`cacheBlock(BlockCacheKey cacheKey, Cacheable cachedItem, boolean inMemory)` Cache the block with the specified name and buffer.
`void`	`cacheBlockWithWait(BlockCacheKey cacheKey, Cacheable cachedItem, boolean inMemory, boolean wait)` Cache the block to ramCache
`protected void`	`cacheBlockWithWaitInternal(BlockCacheKey cacheKey, Cacheable cachedItem, boolean inMemory, boolean wait)`
`private void`	`checkIOErrorIsTolerated()` Check whether we tolerate IO error this time.
`private ByteBuffAllocator.Recycler`	`createRecycler(BucketEntry bucketEntry)` Create the `ByteBuffAllocator.Recycler` for `BucketEntry.refCnt`,which would be used as `RefCnt.recycler` of `HFileBlock.buf` returned from `getBlock(org.apache.hadoop.hbase.io.hfile.BlockCacheKey, boolean, boolean, boolean)`.
`private FileInputStream`	`deleteFileOnClose(File file)` Create an input stream that deletes the file after reading it.
`private void`	`disableCache()` Used to shut down the cache -or- turn it off in the case of something broken.
`(package private) void`	`doDrain(List<BucketCache.RAMQueueEntry> entries, ByteBuffer metaBuff)` Flush the entries in ramCache to IOEngine and add bucket entry to backingMap.
`private boolean`	`doEvictBlock(BlockCacheKey cacheKey, BucketEntry bucketEntry, boolean evictedByEvictionProcess)` Evict the `BlockCacheKey` and `BucketEntry` from `backingMap` and `ramCache`.
`boolean`	`evictBlock(BlockCacheKey cacheKey)` Try to evict the block from `BlockCache` by force.
`boolean`	`evictBlockIfNoRpcReferenced(BlockCacheKey blockCacheKey)` NOTE: This method is only for test.
`int`	`evictBlocksByHfileName(String hfileName)` Evicts all blocks for a specific HFile.
`(package private) boolean`	`evictBucketEntryIfNoRpcReferenced(BlockCacheKey blockCacheKey, BucketEntry bucketEntry)` Evict `BlockCacheKey` and its corresponding `BucketEntry` only if `BucketEntry.isRpcRef()` is false.
`(package private) void`	`freeBucketEntry(BucketEntry bucketEntry)` Free the {`BucketEntry` actually,which could only be invoked when the `BucketEntry.refCnt` becoming 0.
`private void`	`freeEntireBuckets(int completelyFreeBucketsNeeded)` This method will find the buckets that are minimally occupied and are not reference counted and will free them completely without any constraint on the access times of the elements, and as a process will completely free at most the number of buckets passed, sometimes it might not due to changing refCounts
`(package private) void`	`freeSpace(String why)` Free the space if the used size reaches acceptableSize() or one size block couldn't be allocated.
`(package private) float`	`getAcceptableFactor()`
`protected String`	`getAlgorithm()`
`private static String`	`getAllocationFailWarningMessage(BucketAllocatorException fle, BucketCache.RAMQueueEntry re)` Prepare and return a warning message for Bucket Allocator Exception
`BucketAllocator`	`getAllocator()`
`Cacheable`	`getBlock(BlockCacheKey key, boolean caching, boolean repeat, boolean updateCacheMetrics)` Get the buffer of the block with the specified key.
`BlockCache[]`	`getBlockCaches()` Returns The list of sub blockcaches that make up this one; returns null if no sub caches.
`long`	`getBlockCount()` Returns the number of blocks currently cached in the block cache.
`long`	`getCurrentDataSize()` Returns the occupied size of data blocks, in bytes.
`long`	`getCurrentSize()` Returns the occupied size of the block cache, in bytes.
`long`	`getDataBlockCount()` Returns the number of data blocks currently cached in the block cache.
`(package private) float`	`getExtraFreeFactor()`
`long`	`getFreeSize()` Returns the free size of the block cache, in bytes.
`String`	`getIoEngine()`
`private IOEngine`	`getIOEngineFromName(String ioEngineName, long capacity, String persistencePath)` Get the IOEngine from the IO engine name nnn * @return the IOEngine n
`long`	`getMaxSize()` Returns the Max size of the block cache, in bytes.
`(package private) float`	`getMemoryFactor()`
`(package private) float`	`getMinFactor()`
`(package private) float`	`getMultiFactor()`
`(package private) long`	`getPartitionSize(float partitionFactor)`
`(package private) static List<BucketCache.RAMQueueEntry>`	`getRAMQueueEntries(BlockingQueue<BucketCache.RAMQueueEntry> q, List<BucketCache.RAMQueueEntry> receptacle)` Blocks until elements available in `q` then tries to grab as many as possible before returning.
`long`	`getRealCacheSize()`
`int`	`getRpcRefCount(BlockCacheKey cacheKey)`
`(package private) float`	`getSingleFactor()`
`CacheStats`	`getStats()` Get the statistics for this block cache.
`long`	`heapSize()` Return the approximate 'exclusive deep size' of implementing object.
`(package private) boolean`	`isCacheEnabled()`
`Iterator<CachedBlock>`	`iterator()` Returns Iterator over the blocks in the cache.
`private void`	`join()`
`void`	`logStats()`
`private void`	`parsePB(org.apache.hadoop.hbase.shaded.protobuf.generated.BucketCacheProtos.BucketCacheEntry proto)`
`private void`	`persistToFile()`
`protected void`	`putIntoBackingMap(BlockCacheKey key, BucketEntry bucketEntry)` Put the new bucket entry into backingMap.
`protected boolean`	`removeFromRamCache(BlockCacheKey cacheKey)`
`private void`	`retrieveFromFile(int[] bucketSizes)`
`private void`	`sanityCheckConfigs()`
`protected boolean`	`shouldReplaceExistingCacheBlock(BlockCacheKey cacheKey, Cacheable newBlock)`
`void`	`shutdown()` Shutdown the cache.
`long`	`size()` Returns the total size of the block cache, in bytes.
`protected void`	`startWriterThreads()` Called by the constructor to start the writer threads.
`(package private) void`	`stopWriterThreads()` Only used in test n
`private void`	`verifyCapacityAndClasses(long capacitySize, String ioclass, String mapclass)`

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.hbase.io.hfile.BlockCache
isMetaBlock

Methods inherited from interface java.lang.Iterable
forEach, spliterator

- Field Detail
  - LOG
```
private static final org.slf4j.Logger LOG
```
  - SINGLE_FACTOR_CONFIG_NAME
```
static final String SINGLE_FACTOR_CONFIG_NAME
```
    Priority buckets config
    
    See Also:
    
    Constant Field Values
  - MULTI_FACTOR_CONFIG_NAME
```
static final String MULTI_FACTOR_CONFIG_NAME
```
    See Also:
    
    Constant Field Values
  - MEMORY_FACTOR_CONFIG_NAME
```
static final String MEMORY_FACTOR_CONFIG_NAME
```
    See Also:
    
    Constant Field Values
  - EXTRA_FREE_FACTOR_CONFIG_NAME
```
static final String EXTRA_FREE_FACTOR_CONFIG_NAME
```
    See Also:
    
    Constant Field Values
  - ACCEPT_FACTOR_CONFIG_NAME
```
static final String ACCEPT_FACTOR_CONFIG_NAME
```
    See Also:
    
    Constant Field Values
  - MIN_FACTOR_CONFIG_NAME
```
static final String MIN_FACTOR_CONFIG_NAME
```
    See Also:
    
    Constant Field Values
  - DEFAULT_SINGLE_FACTOR
```
static final float DEFAULT_SINGLE_FACTOR
```
    Priority buckets
    
    See Also:
    
    Constant Field Values
  - DEFAULT_MULTI_FACTOR
```
static final float DEFAULT_MULTI_FACTOR
```
    See Also:
    
    Constant Field Values
  - DEFAULT_MEMORY_FACTOR
```
static final float DEFAULT_MEMORY_FACTOR
```
    See Also:
    
    Constant Field Values
  - DEFAULT_MIN_FACTOR
```
static final float DEFAULT_MIN_FACTOR
```
    See Also:
    
    Constant Field Values
  - DEFAULT_EXTRA_FREE_FACTOR
```
private static final float DEFAULT_EXTRA_FREE_FACTOR
```
    See Also:
    
    Constant Field Values
  - DEFAULT_ACCEPT_FACTOR
```
private static final float DEFAULT_ACCEPT_FACTOR
```
    See Also:
    
    Constant Field Values
  - DEFAULT_FREE_ENTIRE_BLOCK_FACTOR
```
private static final int DEFAULT_FREE_ENTIRE_BLOCK_FACTOR
```
    See Also:
    
    Constant Field Values
  - statThreadPeriod
```
private static final int statThreadPeriod
```
    Statistics thread
    
    See Also:
    
    Constant Field Values
  - DEFAULT_WRITER_THREADS
```
static final int DEFAULT_WRITER_THREADS
```
    See Also:
    
    Constant Field Values
  - DEFAULT_WRITER_QUEUE_ITEMS
```
static final int DEFAULT_WRITER_QUEUE_ITEMS
```
    See Also:
    
    Constant Field Values
  - ioEngine
```
final transient IOEngine ioEngine
```
  - ramCache
```
final transient BucketCache.RAMCache ramCache
```
  - backingMap
```
transient ConcurrentHashMap<BlockCacheKey,BucketEntry> backingMap
```
  - cacheEnabled
```
private volatile boolean cacheEnabled
```
    Flag if the cache is enabled or not... We shut it off if there are IO errors for some time, so that Bucket IO exceptions/errors don't bring down the HBase server.
  - writerQueues
```
final transient ArrayList<BlockingQueue<BucketCache.RAMQueueEntry>> writerQueues
```
    A list of writer queues. We have a queue per BucketCache.WriterThread we have running. In other words, the work adding blocks to the BucketCache is divided up amongst the running WriterThreads. Its done by taking hash of the cache key modulo queue count. WriterThread when it runs takes whatever has been recently added and 'drains' the entries to the BucketCache. It then updates the ramCache and backingMap accordingly.
  - writerThreads
```
final transient BucketCache.WriterThread[] writerThreads
```
  - freeInProgress
```
private volatile boolean freeInProgress
```
    Volatile boolean to track if free space is in process or not
  - freeSpaceLock
```
private final transient Lock freeSpaceLock
```
  - realCacheSize
```
private final LongAdder realCacheSize
```
  - heapSize
```
private final LongAdder heapSize
```
  - blockNumber
```
private final LongAdder blockNumber
```
    Current number of cached elements
  - accessCount
```
private final AtomicLong accessCount
```
    Cache access count (sequential ID)
  - DEFAULT_CACHE_WAIT_TIME
```
private static final int DEFAULT_CACHE_WAIT_TIME
```
    See Also:
    
    Constant Field Values
  - wait_when_cache
```
boolean wait_when_cache
```
    Used in tests. If this flag is false and the cache speed is very fast, bucket cache will skip some blocks when caching. If the flag is true, we will wait until blocks are flushed to IOEngine.
  - cacheStats
```
private final BucketCacheStats cacheStats
```
  - persistencePath
```
private final String persistencePath
```
  - cacheCapacity
```
private final long cacheCapacity
```
  - blockSize
```
private final long blockSize
```
    Approximate block size
  - ioErrorsTolerationDuration
```
private final int ioErrorsTolerationDuration
```
    Duration of IO errors tolerated before we disable cache, 1 min as default
  - DEFAULT_ERROR_TOLERATION_DURATION
```
public static final int DEFAULT_ERROR_TOLERATION_DURATION
```
    See Also:
    
    Constant Field Values
  - ioErrorStartTime
```
private volatile long ioErrorStartTime
```
  - offsetLock
```
final transient IdReadWriteLock<Long> offsetLock
```
    A ReentrantReadWriteLock to lock on a particular block identified by offset. The purpose of this is to avoid freeing the block which is being read.
    Key set of offsets in BucketCache is limited so soft reference is the best choice here.
  - blocksByHFile
```
private final NavigableSet<BlockCacheKey> blocksByHFile
```
  - scheduleThreadPool
```
private final transient ScheduledExecutorService scheduleThreadPool
```
    Statistics thread schedule pool (for heavy debugging, could remove)
  - bucketAllocator
```
private transient BucketAllocator bucketAllocator
```
  - acceptableFactor
```
private float acceptableFactor
```
    Acceptable size of cache (no evictions if size < acceptable)
  - minFactor
```
private float minFactor
```
    Minimum threshold of cache (when evicting, evict until size < min)
  - extraFreeFactor
```
private float extraFreeFactor
```
    Free this floating point factor of extra blocks when evicting. For example free the number of blocks requested * (1 + extraFreeFactor)
  - singleFactor
```
private float singleFactor
```
    Single access bucket size
  - multiFactor
```
private float multiFactor
```
    Multiple access bucket size
  - memoryFactor
```
private float memoryFactor
```
    In-memory bucket size
  - FILE_VERIFY_ALGORITHM
```
private static final String FILE_VERIFY_ALGORITHM
```
    See Also:
    
    Constant Field Values
  - DEFAULT_FILE_VERIFY_ALGORITHM
```
private static final String DEFAULT_FILE_VERIFY_ALGORITHM
```
    See Also:
    
    Constant Field Values
  - algorithm
```
private String algorithm
```
    Use MessageDigest class's encryption algorithms to check persistent file integrity, default algorithm is MD5
  - allocFailLogPrevTs
```
private long allocFailLogPrevTs
```
  - ALLOCATION_FAIL_LOG_TIME_PERIOD
```
private static final int ALLOCATION_FAIL_LOG_TIME_PERIOD
```
    See Also:
    
    Constant Field Values
- Constructor Detail
  - BucketCache
```
public BucketCache(String ioEngineName,
                   long capacity,
                   int blockSize,
                   int[] bucketSizes,
                   int writerThreadNum,
                   int writerQLen,
                   String persistencePath)
            throws IOException
```
    Throws:
    
    IOException
  - BucketCache
```
public BucketCache(String ioEngineName,
                   long capacity,
                   int blockSize,
                   int[] bucketSizes,
                   int writerThreadNum,
                   int writerQLen,
                   String persistencePath,
                   int ioErrorsTolerationDuration,
                   org.apache.hadoop.conf.Configuration conf)
            throws IOException
```
    Throws:
    
    IOException
- Method Detail
  - sanityCheckConfigs
```
private void sanityCheckConfigs()
```
  - startWriterThreads
```
protected void startWriterThreads()
```
    Called by the constructor to start the writer threads. Used by tests that need to override starting the threads.
  - isCacheEnabled
```
boolean isCacheEnabled()
```
  - getMaxSize
```
public long getMaxSize()
```
    Description copied from interface: BlockCache
    
    Returns the Max size of the block cache, in bytes.
    
    Specified by:
    
    getMaxSize in interface BlockCache
    
    Returns:
    
    size of cache, in bytes
  - getIoEngine
```
public String getIoEngine()
```
  - getIOEngineFromName
```
private IOEngine getIOEngineFromName(String ioEngineName,
                                     long capacity,
                                     String persistencePath)
                              throws IOException
```
    Get the IOEngine from the IO engine name nnn * @return the IOEngine n
    
    Throws:
    
    IOException
  - cacheBlock
```
public void cacheBlock(BlockCacheKey cacheKey,
                       Cacheable buf)
```
    Cache the block with the specified name and buffer.
    
    Specified by:
    
    cacheBlock in interface BlockCache
    
    Parameters:
    
    cacheKey - block's cache key
    
    buf - block buffer
  - cacheBlock
```
public void cacheBlock(BlockCacheKey cacheKey,
                       Cacheable cachedItem,
                       boolean inMemory)
```
    Cache the block with the specified name and buffer.
    
    Specified by:
    
    cacheBlock in interface BlockCache
    
    Parameters:
    
    cacheKey - block's cache key
    
    cachedItem - block buffer
    
    inMemory - if block is in-memory
  - cacheBlockWithWait
```
public void cacheBlockWithWait(BlockCacheKey cacheKey,
                               Cacheable cachedItem,
                               boolean inMemory,
                               boolean wait)
```
    Cache the block to ramCache
    
    Parameters:
    
    cacheKey - block's cache key
    
    cachedItem - block buffer
    
    inMemory - if block is in-memory
    
    wait - if true, blocking wait when queue is full
  - shouldReplaceExistingCacheBlock
```
protected boolean shouldReplaceExistingCacheBlock(BlockCacheKey cacheKey,
                                                  Cacheable newBlock)
```
  - cacheBlockWithWaitInternal
```
protected void cacheBlockWithWaitInternal(BlockCacheKey cacheKey,
                                          Cacheable cachedItem,
                                          boolean inMemory,
                                          boolean wait)
```
  - getBlock
```
public Cacheable getBlock(BlockCacheKey key,
                          boolean caching,
                          boolean repeat,
                          boolean updateCacheMetrics)
```
    Get the buffer of the block with the specified key.
    
    Specified by:
    
    getBlock in interface BlockCache
    
    Parameters:
    
    key - block's cache key
    
    caching - true if the caller caches blocks on cache misses
    
    repeat - Whether this is a repeat lookup for the same block
    
    updateCacheMetrics - Whether we should update cache metrics or not
    
    Returns:
    
    buffer of specified cache key, or null if not in cache
  - blockEvicted
```
void blockEvicted(BlockCacheKey cacheKey,
                  BucketEntry bucketEntry,
                  boolean decrementBlockNumber,
                  boolean evictedByEvictionProcess)
```
    This method is invoked after the bucketEntry is removed from backingMap
  - freeBucketEntry
```
void freeBucketEntry(BucketEntry bucketEntry)
```
    Free the {BucketEntry actually,which could only be invoked when the BucketEntry.refCnt becoming 0.
  - evictBlock
```
public boolean evictBlock(BlockCacheKey cacheKey)
```
    Try to evict the block from BlockCache by force. We'll call this in few cases:
    1. Close an HFile, and clear all cached blocks.
    2. Call Admin.clearBlockCache(TableName) to clear all blocks for a given table.
    
    Firstly, we'll try to remove the block from RAMCache,and then try to evict from backingMap. Here we evict the block from backingMap immediately, but only free the reference from bucket cache by calling BucketEntry.markedAsEvicted. If there're still some RPC referring this block, block can only be de-allocated when all of them release the block.
    NOTICE: we need to grab the write offset lock firstly before releasing the reference from bucket cache. if we don't, we may read an BucketEntry with refCnt = 0 when getBlock(BlockCacheKey, boolean, boolean, boolean), it's a memory leak.
    
    Specified by:
    
    evictBlock in interface BlockCache
    
    Parameters:
    
    cacheKey - Block to evict
    
    Returns:
    
    true to indicate whether we've evicted successfully or not.
  - doEvictBlock
```
private boolean doEvictBlock(BlockCacheKey cacheKey,
                             BucketEntry bucketEntry,
                             boolean evictedByEvictionProcess)
```
    Evict the BlockCacheKey and BucketEntry from backingMap and ramCache.
    NOTE:When Evict from backingMap,only the matched BlockCacheKey and BucketEntry could be removed.
    
    Parameters:
    
    cacheKey - BlockCacheKey to evict.
    
    bucketEntry - BucketEntry matched BlockCacheKey to evict.
    
    Returns:
    
    true to indicate whether we've evicted successfully or not.
  - createRecycler
```
private ByteBuffAllocator.Recycler createRecycler(BucketEntry bucketEntry)
```
```
 Create the ByteBuffAllocator.Recycler for BucketEntry.refCnt,which would be used as
 RefCnt.recycler of HFileBlock.buf returned from getBlock(org.apache.hadoop.hbase.io.hfile.BlockCacheKey, boolean, boolean, boolean).
 NOTE: for getBlock(org.apache.hadoop.hbase.io.hfile.BlockCacheKey, boolean, boolean, boolean),the RefCnt.recycler of HFileBlock.buf
 from backingMap and ramCache are different:
 1.For RefCnt.recycler of HFileBlock.buf from backingMap,
   it is the return value of current createRecycler(org.apache.hadoop.hbase.io.hfile.bucket.BucketEntry) method.

 2.For RefCnt.recycler of HFileBlock.buf from ramCache,
   it is ByteBuffAllocator.putbackBuffer(java.nio.ByteBuffer).
 
```
  - evictBlockIfNoRpcReferenced
```
public boolean evictBlockIfNoRpcReferenced(BlockCacheKey blockCacheKey)
```
    NOTE: This method is only for test.
  - evictBucketEntryIfNoRpcReferenced
```
boolean evictBucketEntryIfNoRpcReferenced(BlockCacheKey blockCacheKey,
                                          BucketEntry bucketEntry)
```
    Evict BlockCacheKey and its corresponding BucketEntry only if BucketEntry.isRpcRef() is false.
    NOTE:When evict from backingMap,only the matched BlockCacheKey and BucketEntry could be removed.
    
    Parameters:
    
    blockCacheKey - BlockCacheKey to evict.
    
    bucketEntry - BucketEntry matched BlockCacheKey to evict.
    
    Returns:
    
    true to indicate whether we've evicted successfully or not.
  - removeFromRamCache
```
protected boolean removeFromRamCache(BlockCacheKey cacheKey)
```
  - logStats
```
public void logStats()
```
  - getRealCacheSize
```
public long getRealCacheSize()
```
  - acceptableSize
```
public long acceptableSize()
```
  - getPartitionSize
```
long getPartitionSize(float partitionFactor)
```
  - bucketSizesAboveThresholdCount
```
private int bucketSizesAboveThresholdCount(float minFactor)
```
    Return the count of bucketSizeinfos still need free space
  - freeEntireBuckets
```
private void freeEntireBuckets(int completelyFreeBucketsNeeded)
```
    This method will find the buckets that are minimally occupied and are not reference counted and will free them completely without any constraint on the access times of the elements, and as a process will completely free at most the number of buckets passed, sometimes it might not due to changing refCounts
    
    Parameters:
    
    completelyFreeBucketsNeeded - number of buckets to free
  - freeSpace
```
void freeSpace(String why)
```
    Free the space if the used size reaches acceptableSize() or one size block couldn't be allocated. When freeing the space, we use the LRU algorithm and ensure there must be some blocks evicted
    
    Parameters:
    
    why - Why we are being called
  - putIntoBackingMap
```
protected void putIntoBackingMap(BlockCacheKey key,
                                 BucketEntry bucketEntry)
```
    Put the new bucket entry into backingMap. Notice that we are allowed to replace the existing cache with a new block for the same cache key. there's a corner case: one thread cache a block in ramCache, copy to io-engine and add a bucket entry to backingMap. Caching another new block with the same cache key do the same thing for the same cache key, so if not evict the previous bucket entry, then memory leak happen because the previous bucketEntry is gone but the bucketAllocator do not free its memory.
    
    Parameters:
    
    key - Block cache key
    
    bucketEntry - Bucket entry to put into backingMap.
    
    See Also:
    
    BlockCacheUtil.shouldReplaceExistingCacheBlock(BlockCache blockCache,BlockCacheKey cacheKey, Cacheable newBlock)
  - getAllocationFailWarningMessage
```
private static String getAllocationFailWarningMessage(BucketAllocatorException fle,
                                                      BucketCache.RAMQueueEntry re)
```
    Prepare and return a warning message for Bucket Allocator Exception
    
    Parameters:
    
    fle - The exception
    
    re - The RAMQueueEntry for which the exception was thrown.
    
    Returns:
    
    A warning message created from the input RAMQueueEntry object.
  - doDrain
```
void doDrain(List<BucketCache.RAMQueueEntry> entries,
             ByteBuffer metaBuff)
      throws InterruptedException
```
    Flush the entries in ramCache to IOEngine and add bucket entry to backingMap. Process all that are passed in even if failure being sure to remove from ramCache else we'll never undo the references and we'll OOME.
    
    Parameters:
    
    entries - Presumes list passed in here will be processed by this invocation only. No interference expected.
    
    Throws:
    
    InterruptedException
  - getRAMQueueEntries
```
static List<BucketCache.RAMQueueEntry> getRAMQueueEntries(BlockingQueue<BucketCache.RAMQueueEntry> q,
                                                          List<BucketCache.RAMQueueEntry> receptacle)
                                                   throws InterruptedException
```
    Blocks until elements available in q then tries to grab as many as possible before returning.
    
    Parameters:
    
    receptacle - Where to stash the elements taken from queue. We clear before we use it just in case.
    
    q - The queue to take from.
    
    Returns:
    
    receptacle laden with elements taken from the queue or empty if none found.
    
    Throws:
    
    InterruptedException
  - persistToFile
```
private void persistToFile()
                    throws IOException
```
    Throws:
    
    IOException
    
    See Also:
    
    retrieveFromFile(int[])
  - retrieveFromFile
```
private void retrieveFromFile(int[] bucketSizes)
                       throws IOException
```
    Throws:
    
    IOException
    
    See Also:
    
    persistToFile()
  - deleteFileOnClose
```
private FileInputStream deleteFileOnClose(File file)
                                   throws IOException
```
    Create an input stream that deletes the file after reading it. Use in try-with-resources to avoid this pattern where an exception thrown from a finally block may mask earlier exceptions:
```
   File f = ...
   try (FileInputStream fis = new FileInputStream(f)) {
     // use the input stream
   } finally {
     if (!f.delete()) throw new IOException("failed to delete");
   }
 
```
    Parameters:
    
    file - the file to read and delete
    
    Returns:
    
    a FileInputStream for the given file
    
    Throws:
    
    IOException - if there is a problem creating the stream
  - verifyCapacityAndClasses
```
private void verifyCapacityAndClasses(long capacitySize,
                                      String ioclass,
                                      String mapclass)
                               throws IOException
```
    Throws:
    
    IOException
  - parsePB
```
private void parsePB(org.apache.hadoop.hbase.shaded.protobuf.generated.BucketCacheProtos.BucketCacheEntry proto)
              throws IOException
```
    Throws:
    
    IOException
  - checkIOErrorIsTolerated
```
private void checkIOErrorIsTolerated()
```
    Check whether we tolerate IO error this time. If the duration of IOEngine throwing errors exceeds ioErrorsDurationTimeTolerated, we will disable the cache
  - disableCache
```
private void disableCache()
```
    Used to shut down the cache -or- turn it off in the case of something broken.
  - join
```
private void join()
           throws InterruptedException
```
    Throws:
    
    InterruptedException
  - shutdown
```
public void shutdown()
```
    Description copied from interface: BlockCache
    
    Shutdown the cache.
    
    Specified by:
    
    shutdown in interface BlockCache
  - getStats
```
public CacheStats getStats()
```
    Description copied from interface: BlockCache
    
    Get the statistics for this block cache. n
    
    Specified by:
    
    getStats in interface BlockCache
  - getAllocator
```
public BucketAllocator getAllocator()
```
  - heapSize
```
public long heapSize()
```
    Description copied from interface: HeapSize
    
    Return the approximate 'exclusive deep size' of implementing object. Includes count of payload and hosting object sizings.
    
    Specified by:
    
    heapSize in interface HeapSize
  - size
```
public long size()
```
    Description copied from interface: BlockCache
    
    Returns the total size of the block cache, in bytes.
    
    Specified by:
    
    size in interface BlockCache
    
    Returns:
    
    size of cache, in bytes
  - getCurrentDataSize
```
public long getCurrentDataSize()
```
    Description copied from interface: BlockCache
    
    Returns the occupied size of data blocks, in bytes.
    
    Specified by:
    
    getCurrentDataSize in interface BlockCache
    
    Returns:
    
    occupied space in cache, in bytes
  - getFreeSize
```
public long getFreeSize()
```
    Description copied from interface: BlockCache
    
    Returns the free size of the block cache, in bytes.
    
    Specified by:
    
    getFreeSize in interface BlockCache
    
    Returns:
    
    free space in cache, in bytes
  - getBlockCount
```
public long getBlockCount()
```
    Description copied from interface: BlockCache
    
    Returns the number of blocks currently cached in the block cache.
    
    Specified by:
    
    getBlockCount in interface BlockCache
    
    Returns:
    
    number of blocks in the cache
  - getDataBlockCount
```
public long getDataBlockCount()
```
    Description copied from interface: BlockCache
    
    Returns the number of data blocks currently cached in the block cache.
    
    Specified by:
    
    getDataBlockCount in interface BlockCache
    
    Returns:
    
    number of blocks in the cache
  - getCurrentSize
```
public long getCurrentSize()
```
    Description copied from interface: BlockCache
    
    Returns the occupied size of the block cache, in bytes.
    
    Specified by:
    
    getCurrentSize in interface BlockCache
    
    Returns:
    
    occupied space in cache, in bytes
  - getAlgorithm
```
protected String getAlgorithm()
```
  - evictBlocksByHfileName
```
public int evictBlocksByHfileName(String hfileName)
```
    Evicts all blocks for a specific HFile.
    This is used for evict-on-close to remove all blocks of a specific HFile.
    
    Specified by:
    
    evictBlocksByHfileName in interface BlockCache
    
    Returns:
    
    the number of blocks evicted
  - stopWriterThreads
```
void stopWriterThreads()
                throws InterruptedException
```
    Only used in test n
    
    Throws:
    
    InterruptedException
  - iterator
```
public Iterator<CachedBlock> iterator()
```
    Description copied from interface: BlockCache
    
    Returns Iterator over the blocks in the cache.
    
    Specified by:
    
    iterator in interface Iterable<CachedBlock>
    
    Specified by:
    
    iterator in interface BlockCache
  - getBlockCaches
```
public BlockCache[] getBlockCaches()
```
    Description copied from interface: BlockCache
    
    Returns The list of sub blockcaches that make up this one; returns null if no sub caches.
    
    Specified by:
    
    getBlockCaches in interface BlockCache
  - getRpcRefCount
```
public int getRpcRefCount(BlockCacheKey cacheKey)
```
  - getAcceptableFactor
```
float getAcceptableFactor()
```
  - getMinFactor
```
float getMinFactor()
```
  - getExtraFreeFactor
```
float getExtraFreeFactor()
```
  - getSingleFactor
```
float getSingleFactor()
```
  - getMultiFactor
```
float getMultiFactor()
```
  - getMemoryFactor
```
float getMemoryFactor()
```

Class BucketCache

Nested Class Summary

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Methods inherited from interface org.apache.hadoop.hbase.io.hfile.BlockCache

Methods inherited from interface java.lang.Iterable

Field Detail

LOG

SINGLE_FACTOR_CONFIG_NAME

MULTI_FACTOR_CONFIG_NAME

MEMORY_FACTOR_CONFIG_NAME

EXTRA_FREE_FACTOR_CONFIG_NAME

ACCEPT_FACTOR_CONFIG_NAME

MIN_FACTOR_CONFIG_NAME

DEFAULT_SINGLE_FACTOR

DEFAULT_MULTI_FACTOR

DEFAULT_MEMORY_FACTOR

DEFAULT_MIN_FACTOR

DEFAULT_EXTRA_FREE_FACTOR

DEFAULT_ACCEPT_FACTOR

DEFAULT_FREE_ENTIRE_BLOCK_FACTOR

statThreadPeriod

DEFAULT_WRITER_THREADS

DEFAULT_WRITER_QUEUE_ITEMS

ioEngine

ramCache

backingMap

cacheEnabled

writerQueues

writerThreads

freeInProgress

freeSpaceLock

realCacheSize

heapSize

blockNumber

accessCount

DEFAULT_CACHE_WAIT_TIME

wait_when_cache

cacheStats

persistencePath

cacheCapacity

blockSize

ioErrorsTolerationDuration

DEFAULT_ERROR_TOLERATION_DURATION

ioErrorStartTime

offsetLock

blocksByHFile

scheduleThreadPool

bucketAllocator

acceptableFactor

minFactor

extraFreeFactor

singleFactor

multiFactor

memoryFactor

FILE_VERIFY_ALGORITHM

DEFAULT_FILE_VERIFY_ALGORITHM

algorithm

allocFailLogPrevTs

ALLOCATION_FAIL_LOG_TIME_PERIOD

Constructor Detail

BucketCache

BucketCache

Method Detail

sanityCheckConfigs

startWriterThreads

isCacheEnabled

getMaxSize

getIoEngine

getIOEngineFromName

cacheBlock

cacheBlock

cacheBlockWithWait

shouldReplaceExistingCacheBlock

cacheBlockWithWaitInternal

getBlock

blockEvicted

freeBucketEntry