Package org.apache.hadoop.hbase.io.hfile

Class HFileReaderImpl

java.lang.Object

org.apache.hadoop.hbase.io.hfile.HFileReaderImpl

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.conf.Configurable, HFile.CachingBlockReader, HFile.Reader

Direct Known Subclasses:

HFilePreadReader, HFileStreamReader

@Private
public abstract class HFileReaderImpl
extends Object
implements HFile.Reader, org.apache.hadoop.conf.Configurable

Implementation that can handle all hfile versions of HFile.Reader.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	HFileReaderImpl.BlockIndexNotLoadedException
protected static class	HFileReaderImpl.EncodedScanner	Scanner that operates on encoded data blocks.
static class	HFileReaderImpl.HFileScannerImpl
static class	HFileReaderImpl.NotSeekedException	An exception thrown when an operation requiring a scanner to be seeked is invoked on a scanner that is not seeked.

Field Summary

Fields

Modifier and Type	Field	Description
protected final CacheConfig	cacheConf	Block cache configuration.
private org.apache.hadoop.conf.Configuration	conf
protected ReaderContext	context
protected HFileDataBlockEncoder	dataBlockEncoder	What kind of data block encoding should be used while reading, writing, and handling cache.
protected HFileBlockIndex.CellBasedKeyBlockIndexReader	dataBlockIndexReader	Data block index reader keeping the root data index in memory
protected final HFileInfo	fileInfo
protected HFileBlock.FSReader	fsBlockReader	Filesystem-level block reader.
protected HFileContext	hfileContext
static final int	KEY_VALUE_LEN_SIZE	The size of a (key length, value length) tuple that prefixes each entry in a data block.
private static final org.slf4j.Logger	LOG
(package private) static final int	MAX_MINOR_VERSION	Maximum minor version supported by this HFile format
protected HFileBlockIndex.ByteArrayKeyBlockIndexReader	metaBlockIndexReader	Meta block index reader -- always single level
(package private) static final int	MIN_MINOR_VERSION	Minimum minor version supported by this HFile format
static final int	MINOR_VERSION_NO_CHECKSUM	In HFile minor version that does not support checksums
static final int	MINOR_VERSION_WITH_CHECKSUM	Minor versions in HFile starting with this number have hbase checksums
(package private) static final int	MINOR_VERSION_WITH_FAKED_KEY	Minor versions starting with this number have faked index key
protected final String	name	File name to be used for block names
private IdLock	offsetLock	A "sparse lock" implementation allowing to lock on a particular block identified by offset.
protected final org.apache.hadoop.fs.Path	path	Path of file
static final int	PBUF_TRAILER_MINOR_VERSION	HFile minor version that introduced pbuf filetrailer
private final boolean	primaryReplicaReader
protected FixedFileTrailer	trailer

Constructor Summary

Constructors

Constructor	Description
HFileReaderImpl(ReaderContext context, HFileInfo fileInfo, CacheConfig cacheConf, org.apache.hadoop.conf.Configuration conf)	Opens a HFile.

Method Summary

Modifier and Type	Method	Description
void	close()
private DataInput	getBloomFilterMetadata(BlockType blockType)
CacheConfig	getCacheConf()
HFileBlock	getCachedBlock(BlockCacheKey cacheKey, boolean cacheBlock, boolean useLock, boolean updateCacheMetrics, BlockType expectedBlockType, DataBlockEncoding expectedDataBlockEncoding)	Retrieve block from cache.
CellComparator	getComparator()	Returns comparator
Compression.Algorithm	getCompressionAlgorithm()
org.apache.hadoop.conf.Configuration	getConf()
ReaderContext	getContext()
DataBlockEncoding	getDataBlockEncoding()
HFileBlockIndex.CellBasedKeyBlockIndexReader	getDataBlockIndexReader()
DataInput	getDeleteBloomFilterMetadata()	Retrieves delete family Bloom filter metadata as appropriate for each HFile version.
DataBlockEncoding	getEffectiveEncodingInCache(boolean isCompaction)
long	getEntries()	Returns number of KV entries in this HFile
HFileContext	getFileContext()	Return the file context of the HFile this reader belongs to
Optional<ExtendedCell>	getFirstKey()
Optional<byte[]>	getFirstRowKey()	TODO left from HFile version 1: move this to StoreFile after Ryan's patch goes in to eliminate KeyValue here.
DataInput	getGeneralBloomFilterMetadata()	Returns a buffer with the Bloom filter metadata.
HFileInfo	getHFileInfo()
Optional<ExtendedCell>	getLastKey()
Optional<byte[]>	getLastRowKey()	TODO left from HFile version 1: move this to StoreFile after Ryan's patch goes in to eliminate KeyValue here.
int	getMajorVersion()
HFileBlock	getMetaBlock(String metaBlockName, boolean cacheBlock)
HFileBlockIndex.ByteArrayKeyBlockIndexReader	getMetaBlockIndexReader()
String	getName()	Returns this reader's "name".
org.apache.hadoop.fs.Path	getPath()
HFileScanner	getScanner(org.apache.hadoop.conf.Configuration conf, boolean cacheBlocks, boolean pread)	Create a Scanner on this file.
HFileScanner	getScanner(org.apache.hadoop.conf.Configuration conf, boolean cacheBlocks, boolean pread, boolean isCompaction)	Create a Scanner on this file.
FixedFileTrailer	getTrailer()
HFileBlock.FSReader	getUncachedBlockReader()	For testing
boolean	hasMVCCInfo()
long	indexSize()
boolean	isFileInfoLoaded()
boolean	isPrimaryReplicaReader()
long	length()
Optional<ExtendedCell>	midKey()
boolean	prefetchComplete()	Returns false if block prefetching was requested for this file and has not completed, true otherwise
boolean	prefetchStarted()	Returns true if block prefetching was started after waiting for specified delay, false otherwise
HFileBlock	readBlock(long dataBlockOffset, long onDiskBlockSize, boolean cacheBlock, boolean pread, boolean isCompaction, boolean updateCacheMetrics, BlockType expectedBlockType, DataBlockEncoding expectedDataBlockEncoding)	Read in a file block.
HFileBlock	readBlock(long dataBlockOffset, long onDiskBlockSize, boolean cacheBlock, boolean pread, boolean isCompaction, boolean updateCacheMetrics, BlockType expectedBlockType, DataBlockEncoding expectedDataBlockEncoding, boolean cacheOnly)
private void	returnAndEvictBlock(BlockCache cache, BlockCacheKey cacheKey, Cacheable block)
void	setConf(org.apache.hadoop.conf.Configuration conf)
void	setDataBlockEncoder(HFileDataBlockEncoder dataBlockEncoder)
void	setDataBlockIndexReader(HFileBlockIndex.CellBasedKeyBlockIndexReader reader)
void	setMetaBlockIndexReader(HFileBlockIndex.ByteArrayKeyBlockIndexReader reader)
private boolean	shouldUseHeap(BlockType expectedBlockType, boolean cacheBlock)	Whether we use heap or not depends on our intent to cache the block.
String	toString()
private Optional<String>	toStringFirstKey()
private Optional<String>	toStringLastKey()
void	unbufferStream()	To close the stream's socket.
private void	validateBlockType(HFileBlock block, BlockType expectedBlockType)	Compares the actual type of a block retrieved from cache or disk with its expected type and throws an exception in case of a mismatch.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.hadoop.hbase.io.hfile.HFile.Reader

close

Field Details

LOG

private static final org.slf4j.Logger LOG

dataBlockIndexReader

protected HFileBlockIndex.CellBasedKeyBlockIndexReader dataBlockIndexReader

Data block index reader keeping the root data index in memory

metaBlockIndexReader

protected HFileBlockIndex.ByteArrayKeyBlockIndexReader metaBlockIndexReader

Meta block index reader -- always single level

trailer

protected FixedFileTrailer trailer

primaryReplicaReader

private final boolean primaryReplicaReader

dataBlockEncoder

protected HFileDataBlockEncoder dataBlockEncoder

What kind of data block encoding should be used while reading, writing, and handling cache.

cacheConf

protected final CacheConfig cacheConf

Block cache configuration.

context

protected ReaderContext context

fileInfo

protected final HFileInfo fileInfo

path

protected final org.apache.hadoop.fs.Path path

Path of file

name

protected final String name

File name to be used for block names

conf

private org.apache.hadoop.conf.Configuration conf

hfileContext

protected HFileContext hfileContext

fsBlockReader

protected HFileBlock.FSReader fsBlockReader

Filesystem-level block reader.

offsetLock

private IdLock offsetLock

A "sparse lock" implementation allowing to lock on a particular block identified by offset. The purpose of this is to avoid two clients loading the same block, and have all but one client wait to get the block from the cache.

MIN_MINOR_VERSION

static final int MIN_MINOR_VERSION

Minimum minor version supported by this HFile format

See Also:

• Constant Field Values

MAX_MINOR_VERSION

static final int MAX_MINOR_VERSION

Maximum minor version supported by this HFile format

See Also:

• Constant Field Values

MINOR_VERSION_WITH_FAKED_KEY

static final int MINOR_VERSION_WITH_FAKED_KEY

Minor versions starting with this number have faked index key

See Also:

• Constant Field Values

MINOR_VERSION_WITH_CHECKSUM

public static final int MINOR_VERSION_WITH_CHECKSUM

Minor versions in HFile starting with this number have hbase checksums

See Also:

• Constant Field Values

MINOR_VERSION_NO_CHECKSUM

public static final int MINOR_VERSION_NO_CHECKSUM

In HFile minor version that does not support checksums

See Also:

• Constant Field Values

PBUF_TRAILER_MINOR_VERSION

public static final int PBUF_TRAILER_MINOR_VERSION

HFile minor version that introduced pbuf filetrailer

See Also:

• Constant Field Values

KEY_VALUE_LEN_SIZE

public static final int KEY_VALUE_LEN_SIZE

The size of a (key length, value length) tuple that prefixes each entry in a data block.

See Also:

• Constant Field Values

Constructor Details

HFileReaderImpl

public HFileReaderImpl(ReaderContext context,
 HFileInfo fileInfo,
 CacheConfig cacheConf,
 org.apache.hadoop.conf.Configuration conf)
                throws IOException

Opens a HFile.

Parameters:

context - Reader context info

fileInfo - HFile info

cacheConf - Cache configuration.

conf - Configuration

Throws:

IOException

Method Details

getCacheConf

public CacheConfig getCacheConf()

toStringFirstKey

private Optional<String> toStringFirstKey()

toStringLastKey

private Optional<String> toStringLastKey()

toString

public String toString()

Overrides:

toString in class Object

length

public long length()

Specified by:

length in interface HFile.Reader

getFirstKey

public Optional<ExtendedCell> getFirstKey()

Specified by:

getFirstKey in interface HFile.Reader

Returns:

the first key in the file. May be null if file has no entries. Note that this is not the first row key, but rather the byte form of the first KeyValue.

getFirstRowKey

public Optional<byte[]> getFirstRowKey()

TODO left from HFile version 1: move this to StoreFile after Ryan's patch goes in to eliminate KeyValue here.

Specified by:

getFirstRowKey in interface HFile.Reader

Returns:

the first row key, or null if the file is empty.

getLastRowKey

public Optional<byte[]> getLastRowKey()

TODO left from HFile version 1: move this to StoreFile after Ryan's patch goes in to eliminate KeyValue here.

Specified by:

getLastRowKey in interface HFile.Reader

Returns:

the last row key, or null if the file is empty.

getEntries

public long getEntries()

Returns number of KV entries in this HFile

Specified by:

getEntries in interface HFile.Reader

getComparator

public CellComparator getComparator()

Returns comparator

Specified by:

getComparator in interface HFile.Reader

getCompressionAlgorithm

public Compression.Algorithm getCompressionAlgorithm()

indexSize

public long indexSize()

Specified by:

indexSize in interface HFile.Reader

Returns:

the total heap size of data and meta block indexes in bytes. Does not take into account non-root blocks of a multilevel data index.

getName

public String getName()

Description copied from interface: HFile.Reader

Returns this reader's "name". Usually the last component of the path. Needs to be constant as the file is being moved to support caching on write.

Specified by:

getName in interface HFile.Reader

setDataBlockEncoder

public void setDataBlockEncoder(HFileDataBlockEncoder dataBlockEncoder)

Specified by:

setDataBlockEncoder in interface HFile.Reader

setDataBlockIndexReader

public void setDataBlockIndexReader(HFileBlockIndex.CellBasedKeyBlockIndexReader reader)

Specified by:

setDataBlockIndexReader in interface HFile.Reader

getDataBlockIndexReader

public HFileBlockIndex.CellBasedKeyBlockIndexReader getDataBlockIndexReader()

Specified by:

getDataBlockIndexReader in interface HFile.Reader

setMetaBlockIndexReader

public void setMetaBlockIndexReader(HFileBlockIndex.ByteArrayKeyBlockIndexReader reader)

Specified by:

setMetaBlockIndexReader in interface HFile.Reader

getMetaBlockIndexReader

public HFileBlockIndex.ByteArrayKeyBlockIndexReader getMetaBlockIndexReader()

Specified by:

getMetaBlockIndexReader in interface HFile.Reader

getTrailer

public FixedFileTrailer getTrailer()

Specified by:

getTrailer in interface HFile.Reader

getContext

public ReaderContext getContext()

Specified by:

getContext in interface HFile.Reader

getHFileInfo

public HFileInfo getHFileInfo()

Specified by:

getHFileInfo in interface HFile.Reader

isPrimaryReplicaReader

public boolean isPrimaryReplicaReader()

Specified by:

isPrimaryReplicaReader in interface HFile.Reader

getPath

public org.apache.hadoop.fs.Path getPath()

Specified by:

getPath in interface HFile.Reader

getDataBlockEncoding

public DataBlockEncoding getDataBlockEncoding()

Specified by:

getDataBlockEncoding in interface HFile.Reader

getConf

public org.apache.hadoop.conf.Configuration getConf()

Specified by:

getConf in interface org.apache.hadoop.conf.Configurable

setConf

public void setConf(org.apache.hadoop.conf.Configuration conf)

Specified by:

setConf in interface org.apache.hadoop.conf.Configurable

getCachedBlock

@LimitedPrivate("Unittest")
public HFileBlock getCachedBlock(BlockCacheKey cacheKey,
 boolean cacheBlock,
 boolean useLock,
 boolean updateCacheMetrics,
 BlockType expectedBlockType,
 DataBlockEncoding expectedDataBlockEncoding)
                          throws IOException

Retrieve block from cache. Validates the retrieved block's type vs expectedBlockType and its encoding vs. expectedDataBlockEncoding. Unpacks the block as necessary.

Throws:

IOException

returnAndEvictBlock

private void returnAndEvictBlock(BlockCache cache,
 BlockCacheKey cacheKey,
 Cacheable block)

getMetaBlock

public HFileBlock getMetaBlock(String metaBlockName,
 boolean cacheBlock)
                        throws IOException

Specified by:

getMetaBlock in interface HFile.Reader

Parameters:

cacheBlock - Add block to cache, if found

Returns:

block wrapped in a ByteBuffer, with header skipped

Throws:

IOException

shouldUseHeap

private boolean shouldUseHeap(BlockType expectedBlockType,
 boolean cacheBlock)

Whether we use heap or not depends on our intent to cache the block. We want to avoid allocating to off-heap if we intend to cache into the on-heap L1 cache. Otherwise, it's more efficient to allocate to off-heap since we can control GC ourselves for those. So our decision here breaks down as follows:
If block cache is disabled, don't use heap. If we're not using the CombinedBlockCache, use heap unless caching is disabled for the request. Otherwise, only use heap if caching is enabled and the expected block type is not DATA (which goes to off-heap L2 in combined cache).

See Also:

• HFileBlock.FSReader.readBlockData(long, long, boolean, boolean, boolean)

readBlock

public HFileBlock readBlock(long dataBlockOffset,
 long onDiskBlockSize,
 boolean cacheBlock,
 boolean pread,
 boolean isCompaction,
 boolean updateCacheMetrics,
 BlockType expectedBlockType,
 DataBlockEncoding expectedDataBlockEncoding)
                     throws IOException

Description copied from interface: HFile.CachingBlockReader

Read in a file block.

Specified by:

readBlock in interface HFile.CachingBlockReader

Parameters:

dataBlockOffset - offset to read.

onDiskBlockSize - size of the block

isCompaction - is this block being read as part of a compaction

expectedBlockType - the block type we are expecting to read with this read operation, or null to read whatever block type is available and avoid checking (that might reduce caching efficiency of encoded data blocks)

expectedDataBlockEncoding - the data block encoding the caller is expecting data blocks to be in, or null to not perform this check and return the block irrespective of the encoding. This check only applies to data blocks and can be set to null when the caller is expecting to read a non-data block and has set expectedBlockType accordingly.

Returns:

Block wrapped in a ByteBuffer.

Throws:

IOException

readBlock

public HFileBlock readBlock(long dataBlockOffset,
 long onDiskBlockSize,
 boolean cacheBlock,
 boolean pread,
 boolean isCompaction,
 boolean updateCacheMetrics,
 BlockType expectedBlockType,
 DataBlockEncoding expectedDataBlockEncoding,
 boolean cacheOnly)
                     throws IOException

Specified by:

readBlock in interface HFile.CachingBlockReader

Throws:

IOException

hasMVCCInfo

public boolean hasMVCCInfo()

Specified by:

hasMVCCInfo in interface HFile.Reader

validateBlockType

private void validateBlockType(HFileBlock block,
 BlockType expectedBlockType)
                        throws IOException

Compares the actual type of a block retrieved from cache or disk with its expected type and throws an exception in case of a mismatch. Expected block type of BlockType.DATA is considered to match the actual block type [@link BlockType.ENCODED_DATA as well.

Parameters:

block - a block retrieved from cache or disk

expectedBlockType - the expected block type, or null to skip the check

Throws:

IOException

getLastKey

public Optional<ExtendedCell> getLastKey()

Specified by:

getLastKey in interface HFile.Reader

Returns:

Last key as cell in the file. May be null if file has no entries. Note that this is not the last row key, but it is the Cell representation of the last key

midKey

public Optional<ExtendedCell> midKey()
                              throws IOException

Specified by:

midKey in interface HFile.Reader

Returns:

Midkey for this file. We work with block boundaries only so returned midkey is an approximation only.

Throws:

IOException

close

public void close()
           throws IOException

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable

Throws:

IOException

getEffectiveEncodingInCache

public DataBlockEncoding getEffectiveEncodingInCache(boolean isCompaction)

Specified by:

getEffectiveEncodingInCache in interface HFile.Reader

getUncachedBlockReader

public HFileBlock.FSReader getUncachedBlockReader()

For testing

Specified by:

getUncachedBlockReader in interface HFile.Reader

getGeneralBloomFilterMetadata

public DataInput getGeneralBloomFilterMetadata()
                                        throws IOException

Returns a buffer with the Bloom filter metadata. The caller takes ownership of the buffer.

Specified by:

getGeneralBloomFilterMetadata in interface HFile.Reader

Throws:

IOException

getDeleteBloomFilterMetadata

public DataInput getDeleteBloomFilterMetadata()
                                       throws IOException

Description copied from interface: HFile.Reader

Retrieves delete family Bloom filter metadata as appropriate for each HFile version. Knows nothing about how that metadata is structured.

Specified by:

getDeleteBloomFilterMetadata in interface HFile.Reader

Throws:

IOException

getBloomFilterMetadata

private DataInput getBloomFilterMetadata(BlockType blockType)
                                  throws IOException

Throws:

IOException

isFileInfoLoaded

public boolean isFileInfoLoaded()

getFileContext

public HFileContext getFileContext()

Description copied from interface: HFile.Reader

Return the file context of the HFile this reader belongs to

Specified by:

getFileContext in interface HFile.Reader

prefetchComplete

public boolean prefetchComplete()

Returns false if block prefetching was requested for this file and has not completed, true otherwise

Specified by:

prefetchComplete in interface HFile.Reader

prefetchStarted

public boolean prefetchStarted()

Returns true if block prefetching was started after waiting for specified delay, false otherwise

Specified by:

prefetchStarted in interface HFile.Reader

getScanner

public HFileScanner getScanner(org.apache.hadoop.conf.Configuration conf,
 boolean cacheBlocks,
 boolean pread)

Create a Scanner on this file. No seeks or reads are done on creation. Call HFileScanner.seekTo(ExtendedCell) to position an start the read. There is nothing to clean up in a Scanner. Letting go of your references to the scanner is sufficient. NOTE: Do not use this overload of getScanner for compactions. See getScanner(Configuration, boolean, boolean, boolean)

Specified by:

getScanner in interface HFile.Reader

Parameters:

conf - Store configuration.

cacheBlocks - True if we should cache blocks read in by this scanner.

pread - Use positional read rather than seek+read if true (pread is better for random reads, seek+read is better scanning).

Returns:

Scanner on this file.

getScanner

public HFileScanner getScanner(org.apache.hadoop.conf.Configuration conf,
 boolean cacheBlocks,
 boolean pread,
 boolean isCompaction)

Create a Scanner on this file. No seeks or reads are done on creation. Call HFileScanner.seekTo(ExtendedCell) to position an start the read. There is nothing to clean up in a Scanner. Letting go of your references to the scanner is sufficient.

Specified by:

getScanner in interface HFile.Reader

Parameters:

conf - Store configuration.

cacheBlocks - True if we should cache blocks read in by this scanner.

pread - Use positional read rather than seek+read if true (pread is better for random reads, seek+read is better scanning).

isCompaction - is scanner being used for a compaction?

Returns:

Scanner on this file.

getMajorVersion

public int getMajorVersion()

unbufferStream

public void unbufferStream()

Description copied from interface: HFile.Reader

To close the stream's socket. Note: This can be concurrently called from multiple threads and implementation should take care of thread safety.

Specified by:

unbufferStream in interface HFile.Reader