org.apache.hadoop.hbase.io.hfile

Class HFileBlock.FSReaderImpl

java.lang.Object

org.apache.hadoop.hbase.io.hfile.HFileBlock.FSReaderImpl

All Implemented Interfaces:

HFileBlock.FSReader

Enclosing class:

HFileBlock

static class HFileBlock.FSReaderImpl
extends Object
implements HFileBlock.FSReader

Reads version 2 HFile blocks from the filesystem.

Field Summary

Fields

Modifier and Type	Field and Description
private HFileBlockDefaultDecodingContext	defaultDecodingCtx Default context used when BlockType != BlockType.ENCODED_DATA.
private HFileBlockDecodingContext	encodedBlockDecodingCtx
private HFileContext	fileContext
private long	fileSize The size of the file we are reading from, or -1 if unknown.
protected int	hdrSize The size of the header
private HFileSystem	hfs The filesystem used to access data
private String	pathName
private AtomicReference<HFileBlock.PrefetchedHeader>	prefetchedHeader Cache of the NEXT header after this.
private Lock	streamLock
private FSDataInputStreamWrapper	streamWrapper The file system stream of the underlying HFile that does or doesn't do checksum validations in the filesystem

Constructor Summary

Constructors

Constructor and Description
FSReaderImpl(org.apache.hadoop.fs.FSDataInputStream istream, long fileSize, HFileContext fileContext) A constructor that reads files with the latest minor version.
FSReaderImpl(FSDataInputStreamWrapper stream, long fileSize, HFileSystem hfs, org.apache.hadoop.fs.Path path, HFileContext fileContext)

Method Summary

Modifier and Type	Method and Description
HFileBlock.BlockIterator	blockRange(long startOffset, long endOffset) Creates a block iterator over the given portion of the HFile.
private void	cacheNextBlockHeader(long offset, byte[] header, int headerOffset, int headerLength) Save away the next blocks header in atomic reference.
private static int	checkAndGetSizeAsInt(long onDiskSizeWithHeaderL, int hdrSize)
void	closeStreams() Closes the backing streams
HFileBlockDecodingContext	getBlockDecodingContext() Get a decoder for BlockType.ENCODED_DATA blocks from this file.
private ByteBuffer	getCachedHeader(long offset) Check atomic reference cache for this block's header.
HFileBlockDecodingContext	getDefaultBlockDecodingContext() Get the default decoder for blocks from this file.
protected int	readAtOffset(org.apache.hadoop.fs.FSDataInputStream istream, byte[] dest, int destOffset, int size, boolean peekIntoNextBlock, long fileOffset, boolean pread) Does a positional read or a seek and read into the given buffer.
HFileBlock	readBlockData(long offset, long onDiskSizeWithHeaderL, boolean pread, boolean updateMetrics) Reads a version 2 block (version 1 blocks not supported and not expected).
protected HFileBlock	readBlockDataInternal(org.apache.hadoop.fs.FSDataInputStream is, long offset, long onDiskSizeWithHeaderL, boolean pread, boolean verifyChecksum, boolean updateMetrics) Reads a version 2 block.
void	setDataBlockEncoder(HFileDataBlockEncoder encoder)
void	setIncludesMemStoreTS(boolean includesMemstoreTS)
String	toString()
void	unbufferStream() To close the stream's socket.
private boolean	validateChecksum(long offset, ByteBuffer data, int hdrSize) Generates the checksum for the header as well as the data and then validates it.
private void	verifyOnDiskSizeMatchesHeader(int passedIn, ByteBuffer headerBuf, long offset, boolean verifyChecksum) Verify the passed in onDiskSizeWithHeader aligns with what is in the header else something is not right.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

streamWrapper

private FSDataInputStreamWrapper streamWrapper

The file system stream of the underlying HFile that does or doesn't do checksum validations in the filesystem

encodedBlockDecodingCtx

private HFileBlockDecodingContext encodedBlockDecodingCtx

defaultDecodingCtx

private final HFileBlockDefaultDecodingContext defaultDecodingCtx

Default context used when BlockType != BlockType.ENCODED_DATA.

prefetchedHeader

private AtomicReference<HFileBlock.PrefetchedHeader> prefetchedHeader

Cache of the NEXT header after this. Check it is indeed next blocks header before using it. TODO: Review. This overread into next block to fetch next blocks header seems unnecessary given we usually get the block size from the hfile index. Review!

fileSize

private long fileSize

The size of the file we are reading from, or -1 if unknown.

hdrSize

protected final int hdrSize

The size of the header

hfs

private HFileSystem hfs

The filesystem used to access data

fileContext

private HFileContext fileContext

pathName

private String pathName

streamLock

private final Lock streamLock

Constructor Detail

FSReaderImpl

FSReaderImpl(FSDataInputStreamWrapper stream,
             long fileSize,
             HFileSystem hfs,
             org.apache.hadoop.fs.Path path,
             HFileContext fileContext)
      throws IOException

Throws:

IOException

FSReaderImpl

FSReaderImpl(org.apache.hadoop.fs.FSDataInputStream istream,
             long fileSize,
             HFileContext fileContext)
      throws IOException

A constructor that reads files with the latest minor version. This is used by unit tests only.

Throws:

IOException

Method Detail

blockRange

public HFileBlock.BlockIterator blockRange(long startOffset,
                                           long endOffset)

Description copied from interface: HFileBlock.FSReader

Creates a block iterator over the given portion of the HFile. The iterator returns blocks starting with offset such that offset <= startOffset < endOffset. Returned blocks are always unpacked. Used when no hfile index available; e.g. reading in the hfile index blocks themselves on file open.

Specified by:

blockRange in interface HFileBlock.FSReader

Parameters:

startOffset - the offset of the block to start iteration with

endOffset - the offset to end iteration at (exclusive)

Returns:

an iterator of blocks between the two given offsets

readAtOffset

protected int readAtOffset(org.apache.hadoop.fs.FSDataInputStream istream,
                           byte[] dest,
                           int destOffset,
                           int size,
                           boolean peekIntoNextBlock,
                           long fileOffset,
                           boolean pread)
                    throws IOException

Does a positional read or a seek and read into the given buffer. Returns the on-disk size of the next block, or -1 if it could not be read/determined; e.g. EOF.

Parameters:

dest - destination buffer

destOffset - offset into the destination buffer at where to put the bytes we read

size - size of read

peekIntoNextBlock - whether to read the next block's on-disk size

fileOffset - position in the stream to read at

pread - whether we should do a positional read

istream - The input source of data

Returns:

the on-disk size of the next block with header size included, or -1 if it could not be determined; if not -1, the dest INCLUDES the next header

Throws:

IOException

readBlockData

public HFileBlock readBlockData(long offset,
                                long onDiskSizeWithHeaderL,
                                boolean pread,
                                boolean updateMetrics)
                         throws IOException

Reads a version 2 block (version 1 blocks not supported and not expected). Tries to do as little memory allocation as possible, using the provided on-disk size.

Specified by:

readBlockData in interface HFileBlock.FSReader

Parameters:

offset - the offset in the stream to read at

onDiskSizeWithHeaderL - the on-disk size of the block, including the header, or -1 if unknown; i.e. when iterating over blocks reading in the file metadata info.

pread - whether to use a positional read

Returns:

the newly read block

Throws:

IOException

checkAndGetSizeAsInt

private static int checkAndGetSizeAsInt(long onDiskSizeWithHeaderL,
                                        int hdrSize)
                                 throws IOException

Returns:

Check onDiskSizeWithHeaderL size is healthy and then return it as an int

Throws:

IOException

verifyOnDiskSizeMatchesHeader

private void verifyOnDiskSizeMatchesHeader(int passedIn,
                                           ByteBuffer headerBuf,
                                           long offset,
                                           boolean verifyChecksum)
                                    throws IOException

Verify the passed in onDiskSizeWithHeader aligns with what is in the header else something is not right.

Throws:

IOException

getCachedHeader

private ByteBuffer getCachedHeader(long offset)

Check atomic reference cache for this block's header. Cache only good if next read coming through is next in sequence in the block. We read next block's header on the tail of reading the previous block to save a seek. Otherwise, we have to do a seek to read the header before we can pull in the block OR we have to backup the stream because we over-read (the next block's header).

Returns:

The cached block header or null if not found.

See Also:

HFileBlock.PrefetchedHeader, cacheNextBlockHeader(long, byte[], int, int)

cacheNextBlockHeader

private void cacheNextBlockHeader(long offset,
                                  byte[] header,
                                  int headerOffset,
                                  int headerLength)

Save away the next blocks header in atomic reference.

See Also:

getCachedHeader(long), HFileBlock.PrefetchedHeader

readBlockDataInternal

protected HFileBlock readBlockDataInternal(org.apache.hadoop.fs.FSDataInputStream is,
                                           long offset,
                                           long onDiskSizeWithHeaderL,
                                           boolean pread,
                                           boolean verifyChecksum,
                                           boolean updateMetrics)
                                    throws IOException

Reads a version 2 block.

Parameters:

offset - the offset in the stream to read at.

onDiskSizeWithHeaderL - the on-disk size of the block, including the header and checksums if present or -1 if unknown (as a long). Can be -1 if we are doing raw iteration of blocks as when loading up file metadata; i.e. the first read of a new file. Usually non-null gotten from the file index.

pread - whether to use a positional read

verifyChecksum - Whether to use HBase checksums. If HBase checksum is switched off, then use HDFS checksum. Can also flip on/off reading same file if we hit a troublesome patch in an hfile.

Returns:

the HFileBlock or null if there is a HBase checksum mismatch

Throws:

IOException

setIncludesMemStoreTS

public void setIncludesMemStoreTS(boolean includesMemstoreTS)

Specified by:

setIncludesMemStoreTS in interface HFileBlock.FSReader

setDataBlockEncoder

public void setDataBlockEncoder(HFileDataBlockEncoder encoder)

Specified by:

setDataBlockEncoder in interface HFileBlock.FSReader

getBlockDecodingContext

public HFileBlockDecodingContext getBlockDecodingContext()

Description copied from interface: HFileBlock.FSReader

Get a decoder for BlockType.ENCODED_DATA blocks from this file.

Specified by:

getBlockDecodingContext in interface HFileBlock.FSReader

getDefaultBlockDecodingContext

public HFileBlockDecodingContext getDefaultBlockDecodingContext()

Description copied from interface: HFileBlock.FSReader

Get the default decoder for blocks from this file.

Specified by:

getDefaultBlockDecodingContext in interface HFileBlock.FSReader

validateChecksum

private boolean validateChecksum(long offset,
                                 ByteBuffer data,
                                 int hdrSize)
                          throws IOException

Generates the checksum for the header as well as the data and then validates it. If the block doesn't uses checksum, returns false.

Returns:

True if checksum matches, else false.

Throws:

IOException

closeStreams

public void closeStreams()
                  throws IOException

Description copied from interface: HFileBlock.FSReader

Closes the backing streams

Specified by:

closeStreams in interface HFileBlock.FSReader

Throws:

IOException

unbufferStream

public void unbufferStream()

Description copied from interface: HFileBlock.FSReader

To close the stream's socket. Note: This can be concurrently called from multiple threads and implementation should take care of thread safety.

Specified by:

unbufferStream in interface HFileBlock.FSReader

toString

public String toString()

Overrides:

toString in class Object