Package org.apache.hadoop.hbase.io.hfile

package org.apache.hadoop.hbase.io.hfile

Provides implementations of HFile and HFile BlockCache. Caches are configured (and instantiated) by CacheConfig. See head of the CacheConfig class for constants that define cache options and configuration keys to use setting cache options. Cache implementations include the default, native on-heap LruBlockCache and a BucketCache that has a bunch of deploy formats including acting as a L2 for LruBlockCache -- when a block is evicted from LruBlockCache, it goes to the BucketCache and when we search a block, we look in both places -- or, the most common deploy type, using CombinedBlockCache, BucketCache is used as a host for data blocks with meta blocks in an instance of LruBlockCache. BucketCache can also be onheap, offheap, and file-backed.

Which BlockCache should I use?

By default LruBlockCache is on. If you would like to cache more, and offheap (offheap usually means less GC headache), try enabling * BucketCache. Fetching will always be slower when fetching from BucketCache but latencies tend to be less erratic over time (roughly because GC is less). See Nick Dimiduk's BlockCache 101 for some numbers.

Enabling BucketCache

See the HBase Reference Guide Enable BucketCache.

Related Packages

Package	Description
org.apache.hadoop.hbase.io

Class	Description
org.apache.hadoop.hbase.io.hfile.AgeSnapshot	Snapshot of block cache age in cache.
org.apache.hadoop.hbase.io.hfile.BlockCache	Block cache interface.
org.apache.hadoop.hbase.io.hfile.BlockCacheFactory
org.apache.hadoop.hbase.io.hfile.BlockCacheKey	Cache Key for use with implementations of BlockCache
org.apache.hadoop.hbase.io.hfile.BlockCacheUtil	Utilty for aggregating counts in CachedBlocks and toString/toJSON CachedBlocks and BlockCaches.
org.apache.hadoop.hbase.io.hfile.BlockCacheUtil.CachedBlocksByFile	Use one of these to keep a running account of cached blocks by file.
org.apache.hadoop.hbase.io.hfile.BlockCompressedSizePredicator	Allows for defining different compression rate predicates on its implementing classes.
org.apache.hadoop.hbase.io.hfile.BlockIndexChunk
org.apache.hadoop.hbase.io.hfile.BlockPriority
org.apache.hadoop.hbase.io.hfile.BlockType	Various types of HFile blocks.
org.apache.hadoop.hbase.io.hfile.BlockType.BlockCategory
org.apache.hadoop.hbase.io.hfile.BlockWithScanInfo	BlockWithScanInfo is wrapper class for HFileBlock with other attributes.
org.apache.hadoop.hbase.io.hfile.BloomFilterMetrics
org.apache.hadoop.hbase.io.hfile.Cacheable	Cacheable is an interface that allows for an object to be cached.
org.apache.hadoop.hbase.io.hfile.CacheableDeserializer<T extends org.apache.hadoop.hbase.io.hfile.Cacheable>	Interface for a deserializer.
org.apache.hadoop.hbase.io.hfile.CacheableDeserializerIdManager	This class is used to manage the identifiers for CacheableDeserializer.
org.apache.hadoop.hbase.io.hfile.CacheConfig	Stores all of the cache objects and configuration for a single HFile.
org.apache.hadoop.hbase.io.hfile.CachedBlock
org.apache.hadoop.hbase.io.hfile.CacheStats	Class that implements cache metrics.
org.apache.hadoop.hbase.io.hfile.ChecksumUtil	Utility methods to compute and validate checksums.
org.apache.hadoop.hbase.io.hfile.CombinedBlockCache	CombinedBlockCache is an abstraction layer that combines FirstLevelBlockCache and BucketCache.
org.apache.hadoop.hbase.io.hfile.CombinedBlockCache.CombinedCacheStats
org.apache.hadoop.hbase.io.hfile.CompoundBloomFilter	A Bloom filter implementation built on top of BloomFilterChunk, encapsulating a set of fixed-size Bloom filters written out at the time of HFile generation into the data block stream, and loaded on demand at query time.
org.apache.hadoop.hbase.io.hfile.CompoundBloomFilterBase
org.apache.hadoop.hbase.io.hfile.CompoundBloomFilterWriter	Adds methods required for writing a compound Bloom filter to the data section of an HFile to the CompoundBloomFilter class.
org.apache.hadoop.hbase.io.hfile.CorruptHFileException	This exception is thrown when attempts to read an HFile fail due to corruption or truncation issues.
org.apache.hadoop.hbase.io.hfile.ExclusiveMemHFileBlock	The ByteBuffAllocator won't allocate pooled heap ByteBuff now; at the same time, if allocate an off-heap ByteBuff from allocator, then it must be a pooled one.
org.apache.hadoop.hbase.io.hfile.FirstLevelBlockCache	In-memory BlockCache that may be backed by secondary layer(s).
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer	The HFile has a fixed trailer which contains offsets to other variable parts of the file.
org.apache.hadoop.hbase.io.hfile.HFile	File format for hbase.
org.apache.hadoop.hbase.io.hfile.HFile.CachingBlockReader	An abstraction used by the block index.
org.apache.hadoop.hbase.io.hfile.HFile.Reader	An interface used by clients to open and iterate an HFile.
org.apache.hadoop.hbase.io.hfile.HFile.Writer	API required to write an HFile
org.apache.hadoop.hbase.io.hfile.HFile.WriterFactory	This variety of ways to construct writers is used throughout the code, and we want to be able to swap writer implementations.
org.apache.hadoop.hbase.io.hfile.HFileBlock	Cacheable Blocks of an HFile version 2 file.
org.apache.hadoop.hbase.io.hfile.HFileBlock.BlockDeserializer
org.apache.hadoop.hbase.io.hfile.HFileBlockBuilder
org.apache.hadoop.hbase.io.hfile.HFileBlockIndex	Provides functionality to write (HFileBlockIndex.BlockIndexWriter) and read BlockIndexReader single-level and multi-level block indexes.
org.apache.hadoop.hbase.io.hfile.HFileBlockIndex.BlockIndexWriter	Writes the block index into the output stream.
org.apache.hadoop.hbase.io.hfile.HFileContext	Read-only HFile Context Information.
org.apache.hadoop.hbase.io.hfile.HFileContextBuilder	A builder that helps in building up the HFileContext
org.apache.hadoop.hbase.io.hfile.HFileDataBlockEncoder	Controls what kind of data block encoding is used.
org.apache.hadoop.hbase.io.hfile.HFileDataBlockEncoderImpl	Do different kinds of data block encoding according to column family options.
org.apache.hadoop.hbase.io.hfile.HFileIndexBlockEncoder	Controls what kind of index block encoding is used.
org.apache.hadoop.hbase.io.hfile.HFileIndexBlockEncoder.EncodedSeeker
org.apache.hadoop.hbase.io.hfile.HFileIndexBlockEncoderImpl	Do different kinds of index block encoding according to column family options.
org.apache.hadoop.hbase.io.hfile.HFileInfo	Metadata Map of attributes for HFile written out as HFile Trailer.
org.apache.hadoop.hbase.io.hfile.HFilePreadReader	Implementation of HFile.Reader to deal with pread.
org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter	Implements pretty-printing functionality for HFiles.
org.apache.hadoop.hbase.io.hfile.HFileReaderImpl	Implementation that can handle all hfile versions of HFile.Reader.
org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.BlockIndexNotLoadedException
org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.EncodedScanner	Scanner that operates on encoded data blocks.
org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.HFileScannerImpl
org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.NotSeekedException	An exception thrown when an operation requiring a scanner to be seeked is invoked on a scanner that is not seeked.
org.apache.hadoop.hbase.io.hfile.HFileScanner	A scanner allows you to position yourself within a HFile and scan through it.
org.apache.hadoop.hbase.io.hfile.HFileStreamReader	Implementation of HFile.Reader to deal with stream read do not perform any prefetch operations (HFilePreadReader will do this).
org.apache.hadoop.hbase.io.hfile.HFileWriterImpl	Common functionality needed by all versions of HFile writers.
org.apache.hadoop.hbase.io.hfile.InclusiveCombinedBlockCache
org.apache.hadoop.hbase.io.hfile.IndexOnlyLruBlockCache	An on heap block cache implementation extended LruBlockCache and only cache index block.
org.apache.hadoop.hbase.io.hfile.InlineBlockWriter	A way to write "inline" blocks into an HFile.
org.apache.hadoop.hbase.io.hfile.InvalidHFileException	Thrown when an invalid HFile format is detected
org.apache.hadoop.hbase.io.hfile.LruAdaptiveBlockCache	This realisation improve performance of classical LRU cache up to 3 times via reduce GC job.
org.apache.hadoop.hbase.io.hfile.LruBlockCache	A block cache implementation that is memory-aware using HeapSize, memory-bound using an LRU eviction algorithm, and concurrent: backed by a ConcurrentHashMap and with a non-blocking eviction thread giving constant-time LruBlockCache.cacheBlock(org.apache.hadoop.hbase.io.hfile.BlockCacheKey, org.apache.hadoop.hbase.io.hfile.Cacheable, boolean) and LruBlockCache.getBlock(org.apache.hadoop.hbase.io.hfile.BlockCacheKey, boolean, boolean, boolean) operations.
org.apache.hadoop.hbase.io.hfile.LruCachedBlock	Represents an entry in the LruBlockCache.
org.apache.hadoop.hbase.io.hfile.LruCachedBlockQueue	A memory-bound queue that will grow until an element brings total size >= maxSize.
org.apache.hadoop.hbase.io.hfile.MemcachedBlockCache	Class to store blocks into memcached.
org.apache.hadoop.hbase.io.hfile.NoOpDataBlockEncoder	Does not perform any kind of encoding/decoding.
org.apache.hadoop.hbase.io.hfile.NoOpIndexBlockEncoder	Does not perform any kind of encoding/decoding.
org.apache.hadoop.hbase.io.hfile.NoOpIndexBlockEncoder.NoOpEncodedSeeker
org.apache.hadoop.hbase.io.hfile.PrefetchExecutor
org.apache.hadoop.hbase.io.hfile.PreviousBlockCompressionRatePredicator	This BlockCompressedSizePredicator implementation adjusts the block size limit based on the compression rate of the block contents read so far.
org.apache.hadoop.hbase.io.hfile.ReaderContext	Carries the information on some of the meta data about the HFile Reader
org.apache.hadoop.hbase.io.hfile.ReaderContext.ReaderType
org.apache.hadoop.hbase.io.hfile.ReaderContextBuilder	A builder that helps in building up the ReaderContext
org.apache.hadoop.hbase.io.hfile.ResizableBlockCache	BlockCache which is resizable.
org.apache.hadoop.hbase.io.hfile.SharedMemHFileBlock	The ByteBuffAllocator won't allocate pooled heap ByteBuff now; at the same time, if allocate an off-heap ByteBuff from allocator, then it must be a pooled one.
org.apache.hadoop.hbase.io.hfile.TinyLfuBlockCache	A block cache that is memory-aware using HeapSize, memory bounded using the W-TinyLFU eviction algorithm, and concurrent.
org.apache.hadoop.hbase.io.hfile.UncompressedBlockSizePredicator	This BlockCompressedSizePredicator implementation doesn't actually performs any predicate and simply returns true on shouldFinishBlock.