org.apache.hadoop.hbase.io.hfile

Class HFileBlock.FSReaderImpl

java.lang.Object

org.apache.hadoop.hbase.io.hfile.HFileBlock.FSReaderImpl

All Implemented Interfaces:

HFileBlock.FSReader

Enclosing class:

HFileBlock

static class HFileBlock.FSReaderImpl
extends Object
implements HFileBlock.FSReader

Reads version 2 HFile blocks from the filesystem.

Field Summary

Fields

Modifier and Type	Field and Description
private ByteBuffAllocator	allocator
private HFileBlockDefaultDecodingContext	defaultDecodingCtx Default context used when BlockType != BlockType.ENCODED_DATA.
private HFileBlockDecodingContext	encodedBlockDecodingCtx
private HFileContext	fileContext
private long	fileSize The size of the file we are reading from, or -1 if unknown.
protected int	hdrSize The size of the header
private HFileSystem	hfs The filesystem used to access data
private boolean	isPreadAllBytes
private String	pathName
private AtomicReference<HFileBlock.PrefetchedHeader>	prefetchedHeader Cache of the NEXT header after this.
private Lock	streamLock
private FSDataInputStreamWrapper	streamWrapper The file system stream of the underlying HFile that does or doesn't do checksum validations in the filesystem

Constructor Summary

Constructors

Constructor and Description
FSReaderImpl(ReaderContext readerContext, HFileContext fileContext, ByteBuffAllocator allocator, org.apache.hadoop.conf.Configuration conf)

Method Summary

Modifier and Type	Method and Description
private ByteBuff	allocate(int size, boolean intoHeap)
HFileBlock.BlockIterator	blockRange(long startOffset, long endOffset) Creates a block iterator over the given portion of the HFile.
private void	cacheNextBlockHeader(long offset, ByteBuff onDiskBlock, int onDiskSizeWithHeader, int headerLength) Save away the next blocks header in atomic reference.
private static int	checkAndGetSizeAsInt(long onDiskSizeWithHeaderL, int hdrSize) Returns Check onDiskSizeWithHeaderL size is healthy and then return it as an int
void	closeStreams() Closes the backing streams
HFileBlockDecodingContext	getBlockDecodingContext() Get a decoder for BlockType.ENCODED_DATA blocks from this file.
private ByteBuff	getCachedHeader(long offset) Check atomic reference cache for this block's header.
HFileBlockDecodingContext	getDefaultBlockDecodingContext() Get the default decoder for blocks from this file.
private int	getNextBlockOnDiskSize(boolean readNextHeader, ByteBuff onDiskBlock, int onDiskSizeWithHeader)
protected boolean	readAtOffset(org.apache.hadoop.fs.FSDataInputStream istream, ByteBuff dest, int size, boolean peekIntoNextBlock, long fileOffset, boolean pread) Does a positional read or a seek and read into the given byte buffer.
HFileBlock	readBlockData(long offset, long onDiskSizeWithHeaderL, boolean pread, boolean updateMetrics, boolean intoHeap) Reads a version 2 block (version 1 blocks not supported and not expected).
protected HFileBlock	readBlockDataInternal(org.apache.hadoop.fs.FSDataInputStream is, long offset, long onDiskSizeWithHeaderL, boolean pread, boolean verifyChecksum, boolean updateMetrics, boolean intoHeap) Reads a version 2 block.
void	setDataBlockEncoder(HFileDataBlockEncoder encoder, org.apache.hadoop.conf.Configuration conf)
void	setIncludesMemStoreTS(boolean includesMemstoreTS)
String	toString()
void	unbufferStream() To close the stream's socket.
private boolean	validateChecksum(long offset, ByteBuff data, int hdrSize) Generates the checksum for the header as well as the data and then validates it.
private void	verifyOnDiskSizeMatchesHeader(int passedIn, ByteBuff headerBuf, long offset, boolean verifyChecksum) Verify the passed in onDiskSizeWithHeader aligns with what is in the header else something is not right.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

streamWrapper

private FSDataInputStreamWrapper streamWrapper

The file system stream of the underlying HFile that does or doesn't do checksum validations in the filesystem

encodedBlockDecodingCtx

private HFileBlockDecodingContext encodedBlockDecodingCtx

defaultDecodingCtx

private final HFileBlockDefaultDecodingContext defaultDecodingCtx

Default context used when BlockType != BlockType.ENCODED_DATA.

prefetchedHeader

private AtomicReference<HFileBlock.PrefetchedHeader> prefetchedHeader

Cache of the NEXT header after this. Check it is indeed next blocks header before using it. TODO: Review. This overread into next block to fetch next blocks header seems unnecessary given we usually get the block size from the hfile index. Review!

fileSize

private long fileSize

The size of the file we are reading from, or -1 if unknown.

hdrSize

protected final int hdrSize

The size of the header

hfs

private HFileSystem hfs

The filesystem used to access data

fileContext

private HFileContext fileContext

pathName

private String pathName

allocator

private final ByteBuffAllocator allocator

streamLock

private final Lock streamLock

isPreadAllBytes

private final boolean isPreadAllBytes

Constructor Detail

FSReaderImpl

FSReaderImpl(ReaderContext readerContext,
             HFileContext fileContext,
             ByteBuffAllocator allocator,
             org.apache.hadoop.conf.Configuration conf)
      throws IOException

Throws:

IOException

Method Detail

blockRange

public HFileBlock.BlockIterator blockRange(long startOffset,
                                           long endOffset)

Description copied from interface: HFileBlock.FSReader

Creates a block iterator over the given portion of the HFile. The iterator returns blocks starting with offset such that offset <= startOffset < endOffset. Returned blocks are always unpacked. Used when no hfile index available; e.g. reading in the hfile index blocks themselves on file open.

Specified by:

blockRange in interface HFileBlock.FSReader

Parameters:

startOffset - the offset of the block to start iteration with

endOffset - the offset to end iteration at (exclusive)

Returns:

an iterator of blocks between the two given offsets

readAtOffset

protected boolean readAtOffset(org.apache.hadoop.fs.FSDataInputStream istream,
                               ByteBuff dest,
                               int size,
                               boolean peekIntoNextBlock,
                               long fileOffset,
                               boolean pread)
                        throws IOException

Does a positional read or a seek and read into the given byte buffer. We need take care that we will call the ByteBuff.release() for every exit to deallocate the ByteBuffers, otherwise the memory leak may happen.

Parameters:

dest - destination buffer

size - size of read

peekIntoNextBlock - whether to read the next block's on-disk size

fileOffset - position in the stream to read at

pread - whether we should do a positional read

istream - The input source of data

Returns:

true to indicate the destination buffer include the next block header, otherwise only include the current block data without the next block header.

Throws:

IOException - if any IO error happen.

readBlockData

public HFileBlock readBlockData(long offset,
                                long onDiskSizeWithHeaderL,
                                boolean pread,
                                boolean updateMetrics,
                                boolean intoHeap)
                         throws IOException

Reads a version 2 block (version 1 blocks not supported and not expected). Tries to do as little memory allocation as possible, using the provided on-disk size.

Specified by:

readBlockData in interface HFileBlock.FSReader

Parameters:

offset - the offset in the stream to read at

onDiskSizeWithHeaderL - the on-disk size of the block, including the header, or -1 if unknown; i.e. when iterating over blocks reading in the file metadata info.

pread - whether to use a positional read

updateMetrics - whether to update the metrics

intoHeap - allocate ByteBuff of block from heap or off-heap.

Returns:

the newly read block

Throws:

IOException

See Also:

for more details about the useHeap.

checkAndGetSizeAsInt

private static int checkAndGetSizeAsInt(long onDiskSizeWithHeaderL,
                                        int hdrSize)
                                 throws IOException

Returns Check onDiskSizeWithHeaderL size is healthy and then return it as an int

Throws:

IOException

verifyOnDiskSizeMatchesHeader

private void verifyOnDiskSizeMatchesHeader(int passedIn,
                                           ByteBuff headerBuf,
                                           long offset,
                                           boolean verifyChecksum)
                                    throws IOException

Verify the passed in onDiskSizeWithHeader aligns with what is in the header else something is not right.

Throws:

IOException

getCachedHeader

private ByteBuff getCachedHeader(long offset)

Check atomic reference cache for this block's header. Cache only good if next read coming through is next in sequence in the block. We read next block's header on the tail of reading the previous block to save a seek. Otherwise, we have to do a seek to read the header before we can pull in the block OR we have to backup the stream because we over-read (the next block's header).

Returns:

The cached block header or null if not found.

See Also:

HFileBlock.PrefetchedHeader, cacheNextBlockHeader(long, ByteBuff, int, int)

cacheNextBlockHeader

private void cacheNextBlockHeader(long offset,
                                  ByteBuff onDiskBlock,
                                  int onDiskSizeWithHeader,
                                  int headerLength)

Save away the next blocks header in atomic reference.

See Also:

getCachedHeader(long), HFileBlock.PrefetchedHeader

getNextBlockOnDiskSize

private int getNextBlockOnDiskSize(boolean readNextHeader,
                                   ByteBuff onDiskBlock,
                                   int onDiskSizeWithHeader)

allocate

private ByteBuff allocate(int size,
                          boolean intoHeap)

readBlockDataInternal

protected HFileBlock readBlockDataInternal(org.apache.hadoop.fs.FSDataInputStream is,
                                           long offset,
                                           long onDiskSizeWithHeaderL,
                                           boolean pread,
                                           boolean verifyChecksum,
                                           boolean updateMetrics,
                                           boolean intoHeap)
                                    throws IOException

Reads a version 2 block.

Parameters:

offset - the offset in the stream to read at.

onDiskSizeWithHeaderL - the on-disk size of the block, including the header and checksums if present or -1 if unknown (as a long). Can be -1 if we are doing raw iteration of blocks as when loading up file metadata; i.e. the first read of a new file. Usually non-null gotten from the file index.

pread - whether to use a positional read

verifyChecksum - Whether to use HBase checksums. If HBase checksum is switched off, then use HDFS checksum. Can also flip on/off reading same file if we hit a troublesome patch in an hfile.

updateMetrics - whether need to update the metrics.

intoHeap - allocate the ByteBuff of block from heap or off-heap.

Returns:

the HFileBlock or null if there is a HBase checksum mismatch

Throws:

IOException

setIncludesMemStoreTS

public void setIncludesMemStoreTS(boolean includesMemstoreTS)

Specified by:

setIncludesMemStoreTS in interface HFileBlock.FSReader

setDataBlockEncoder

public void setDataBlockEncoder(HFileDataBlockEncoder encoder,
                                org.apache.hadoop.conf.Configuration conf)

Specified by:

setDataBlockEncoder in interface HFileBlock.FSReader

getBlockDecodingContext

public HFileBlockDecodingContext getBlockDecodingContext()

Description copied from interface: HFileBlock.FSReader

Get a decoder for BlockType.ENCODED_DATA blocks from this file.

Specified by:

getBlockDecodingContext in interface HFileBlock.FSReader

getDefaultBlockDecodingContext

public HFileBlockDecodingContext getDefaultBlockDecodingContext()

Description copied from interface: HFileBlock.FSReader

Get the default decoder for blocks from this file.

Specified by:

getDefaultBlockDecodingContext in interface HFileBlock.FSReader

validateChecksum

private boolean validateChecksum(long offset,
                                 ByteBuff data,
                                 int hdrSize)

Generates the checksum for the header as well as the data and then validates it. If the block doesn't uses checksum, returns false.

Returns:

True if checksum matches, else false.

closeStreams

public void closeStreams()
                  throws IOException

Description copied from interface: HFileBlock.FSReader

Closes the backing streams

Specified by:

closeStreams in interface HFileBlock.FSReader

Throws:

IOException

unbufferStream

public void unbufferStream()

Description copied from interface: HFileBlock.FSReader

To close the stream's socket. Note: This can be concurrently called from multiple threads and implementation should take care of thread safety.

Specified by:

unbufferStream in interface HFileBlock.FSReader

toString

public String toString()

Overrides:

toString in class Object