org.apache.hadoop.hbase.io.hfile

Class HFile

java.lang.Object

org.apache.hadoop.hbase.io.hfile.HFile

@InterfaceAudience.Private
public class HFile
extends Object

File format for hbase. A file of sorted key/value pairs. Both keys and values are byte arrays.

The memory footprint of a HFile includes the following (below is taken from the TFile documentation but applies also to HFile):

• Some constant overhead of reading or writing a compressed block.
  • Each compressed block requires one compression/decompression codec for I/O.
  • Temporary space to buffer the key.
  • Temporary space to buffer the value.
• HFile index, which is proportional to the total number of Data Blocks. The total amount of memory needed to hold the index can be estimated as (56+AvgKeySize)*NumBlocks.

Suggestions on performance optimization.

• Minimum block size. We recommend a setting of minimum block size between 8KB to 1MB for general usage. Larger block size is preferred if files are primarily for sequential access. However, it would lead to inefficient random access (because there are more data to decompress). Smaller blocks are good for random access, but require more memory to hold the block index, and may be slower to create (because we must flush the compressor stream at the conclusion of each data block, which leads to an FS I/O flush). Further, due to the internal caching in Compression codec, the smallest possible block size would be around 20KB-30KB.
• The current implementation does not offer true multi-threading for reading. The implementation uses FSDataInputStream seek()+read(), which is shown to be much faster than positioned-read call in single thread mode. However, it also means that if multiple threads attempt to access the same HFile (using multiple scanners) simultaneously, the actual I/O is carried out sequentially even if they access different DFS blocks (Reexamine! pread seems to be 10% faster than seek+read in my testing -- stack).
• Compression codec. Use "none" if the data is not very compressable (by compressable, I mean a compression ratio at least 2:1). Generally, use "lzo" as the starting point for experimenting. "gz" overs slightly better compression ratio over "lzo" but requires 4x CPU to compress and 2x CPU to decompress, comparing to "lzo".

For more on the background behind HFile, see HBASE-61.

File is made of data blocks followed by meta data blocks (if any), a fileinfo block, data block index, meta data block index, and a fixed size trailer which records the offsets at which file changes content type.

<data blocks><meta blocks><fileinfo><
 data index><meta index><trailer>

Each block has a bit of magic at its start. Block are comprised of key/values. In data blocks, they are both byte arrays. Metadata blocks are a String key and a byte array value. An empty file looks like this:

<fileinfo><trailer>

. That is, there are not data nor meta blocks present.

TODO: Do scanners need to be able to take a start and end row? TODO: Should BlockIndex know the name of its file? Should it have a Path that points at its file say for the case where an index lives apart from an HFile instance?

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	HFile.CachingBlockReader An abstraction used by the block index.
static class	HFile.FileInfo Metadata for this file.
static interface	HFile.Reader An interface used by clients to open and iterate an HFile.
static interface	HFile.Writer API required to write an HFile
static class	HFile.WriterFactory This variety of ways to construct writers is used throughout the code, and we want to be able to swap writer implementations.

Field Summary

Fields

Modifier and Type	Field and Description
static String	BLOOM_FILTER_DATA_KEY Meta data block name for bloom filter bits.
(package private) static LongAdder	CHECKSUM_FAILURES
static LongAdder	DATABLOCK_READ_COUNT
static int	DEFAULT_BYTES_PER_CHECKSUM The number of bytes per checksum.
static String	DEFAULT_COMPRESSION Default compression name: none.
static Compression.Algorithm	DEFAULT_COMPRESSION_ALGORITHM Default compression: none.
static String	FORMAT_VERSION_KEY The configuration key for HFile version to use for new files
(package private) static org.slf4j.Logger	LOG
static int	MAX_FORMAT_VERSION Maximum supported HFile format version
static int	MAXIMUM_KEY_LENGTH Maximum length of key in HFile.
(package private) static MetricsIO	metrics Static instance for the metrics so that HFileReaders access the same instance
static int	MIN_FORMAT_VERSION Minimum supported HFile format version
static int	MIN_FORMAT_VERSION_WITH_TAGS Minimum HFile format version with support for persisting cell tags
static int	MIN_NUM_HFILE_PATH_LEVELS We assume that HFile path ends with ROOT_DIR/TABLE_NAME/REGION_NAME/CF_NAME/HFILE, so it has at least this many levels of nesting.

Constructor Summary

Constructors

Constructor and Description
HFile()

Method Summary

Modifier and Type	Method and Description
static void	checkFormatVersion(int version) Checks the given HFile format version, and throws an exception if invalid.
static void	checkHFileVersion(org.apache.hadoop.conf.Configuration c)
static HFile.Reader	createReader(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, CacheConfig cacheConf, boolean primaryReplicaReader, org.apache.hadoop.conf.Configuration conf)
static HFile.Reader	createReader(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration conf) Creates reader with cache configuration disabled
static HFile.Reader	createReader(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, FSDataInputStreamWrapper fsdis, long size, CacheConfig cacheConf, boolean primaryReplicaReader, org.apache.hadoop.conf.Configuration conf) The sockets and the file descriptors held by the method parameter FSDataInputStreamWrapper passed will be freed after its usage so caller needs to ensure that no other threads have access to the same passed reference.
(package private) static HFile.Reader	createReaderFromStream(org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.FSDataInputStream fsdis, long size, CacheConfig cacheConf, org.apache.hadoop.conf.Configuration conf) This factory method is used only by unit tests.
static long	getAndResetChecksumFailuresCount() Number of checksum verification failures.
static long	getChecksumFailuresCount() Number of checksum verification failures.
static int	getFormatVersion(org.apache.hadoop.conf.Configuration conf)
(package private) static List<org.apache.hadoop.fs.Path>	getStoreFiles(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path regionDir) Returns all HFiles belonging to the given region directory.
static String[]	getSupportedCompressionAlgorithms() Get names of supported compression algorithms.
static HFile.WriterFactory	getWriterFactory(org.apache.hadoop.conf.Configuration conf, CacheConfig cacheConf) Returns the factory to be used to create HFile writers
static HFile.WriterFactory	getWriterFactoryNoCache(org.apache.hadoop.conf.Configuration conf) Returns the factory to be used to create HFile writers.
static boolean	isHFileFormat(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.FileStatus fileStatus) Returns true if the specified file has a valid HFile Trailer.
static boolean	isHFileFormat(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path) Returns true if the specified file has a valid HFile Trailer.
static boolean	isReservedFileInfoKey(byte[] key) Return true if the given file info key is reserved for internal use.
(package private) static int	longToInt(long l)
static void	main(String[] args)
private static HFile.Reader	openReader(org.apache.hadoop.fs.Path path, FSDataInputStreamWrapper fsdis, long size, CacheConfig cacheConf, HFileSystem hfs, boolean primaryReplicaReader, org.apache.hadoop.conf.Configuration conf) Method returns the reader given the specified arguments.
static void	updateReadLatency(long latencyMillis, boolean pread)
static void	updateWriteLatency(long latencyMillis)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

static final org.slf4j.Logger LOG

MAXIMUM_KEY_LENGTH

public static final int MAXIMUM_KEY_LENGTH

Maximum length of key in HFile.

See Also:

Constant Field Values

DEFAULT_COMPRESSION_ALGORITHM

public static final Compression.Algorithm DEFAULT_COMPRESSION_ALGORITHM

Default compression: none.

MIN_FORMAT_VERSION

public static final int MIN_FORMAT_VERSION

Minimum supported HFile format version

See Also:

Constant Field Values

MAX_FORMAT_VERSION

public static final int MAX_FORMAT_VERSION

Maximum supported HFile format version

See Also:

Constant Field Values

MIN_FORMAT_VERSION_WITH_TAGS

public static final int MIN_FORMAT_VERSION_WITH_TAGS

Minimum HFile format version with support for persisting cell tags

See Also:

Constant Field Values

DEFAULT_COMPRESSION

public static final String DEFAULT_COMPRESSION

Default compression name: none.

BLOOM_FILTER_DATA_KEY

public static final String BLOOM_FILTER_DATA_KEY

Meta data block name for bloom filter bits.

See Also:

Constant Field Values

MIN_NUM_HFILE_PATH_LEVELS

public static final int MIN_NUM_HFILE_PATH_LEVELS

We assume that HFile path ends with ROOT_DIR/TABLE_NAME/REGION_NAME/CF_NAME/HFILE, so it has at least this many levels of nesting. This is needed for identifying table and CF name from an HFile path.

See Also:

Constant Field Values

DEFAULT_BYTES_PER_CHECKSUM

public static final int DEFAULT_BYTES_PER_CHECKSUM

The number of bytes per checksum.

See Also:

Constant Field Values

CHECKSUM_FAILURES

static final LongAdder CHECKSUM_FAILURES

DATABLOCK_READ_COUNT

public static final LongAdder DATABLOCK_READ_COUNT

metrics

static final MetricsIO metrics

Static instance for the metrics so that HFileReaders access the same instance

FORMAT_VERSION_KEY

public static final String FORMAT_VERSION_KEY

The configuration key for HFile version to use for new files

See Also:

Constant Field Values

Constructor Detail

HFile

public HFile()

Method Detail

getAndResetChecksumFailuresCount

public static final long getAndResetChecksumFailuresCount()

Number of checksum verification failures. It also clears the counter.

getChecksumFailuresCount

public static final long getChecksumFailuresCount()

Number of checksum verification failures. It also clears the counter.

updateReadLatency

public static final void updateReadLatency(long latencyMillis,
                                           boolean pread)

updateWriteLatency

public static final void updateWriteLatency(long latencyMillis)

getFormatVersion

public static int getFormatVersion(org.apache.hadoop.conf.Configuration conf)

getWriterFactoryNoCache

public static final HFile.WriterFactory getWriterFactoryNoCache(org.apache.hadoop.conf.Configuration conf)

Returns the factory to be used to create HFile writers. Disables block cache access for all writers created through the returned factory.

getWriterFactory

public static final HFile.WriterFactory getWriterFactory(org.apache.hadoop.conf.Configuration conf,
                                                         CacheConfig cacheConf)

Returns the factory to be used to create HFile writers

openReader

private static HFile.Reader openReader(org.apache.hadoop.fs.Path path,
                                       FSDataInputStreamWrapper fsdis,
                                       long size,
                                       CacheConfig cacheConf,
                                       HFileSystem hfs,
                                       boolean primaryReplicaReader,
                                       org.apache.hadoop.conf.Configuration conf)
                                throws IOException

Method returns the reader given the specified arguments. TODO This is a bad abstraction. See HBASE-6635.

Parameters:

path - hfile's path

fsdis - stream of path's file

size - max size of the trailer.

cacheConf - Cache configuation values, cannot be null.

hfs -

primaryReplicaReader - true if this is a reader for primary replica

Returns:

an appropriate instance of HFileReader

Throws:

IOException - If file is invalid, will throw CorruptHFileException flavored IOException

createReader

public static HFile.Reader createReader(org.apache.hadoop.fs.FileSystem fs,
                                        org.apache.hadoop.fs.Path path,
                                        FSDataInputStreamWrapper fsdis,
                                        long size,
                                        CacheConfig cacheConf,
                                        boolean primaryReplicaReader,
                                        org.apache.hadoop.conf.Configuration conf)
                                 throws IOException

The sockets and the file descriptors held by the method parameter FSDataInputStreamWrapper passed will be freed after its usage so caller needs to ensure that no other threads have access to the same passed reference.

Parameters:

fs - A file system

path - Path to HFile

fsdis - a stream of path's file

size - max size of the trailer.

cacheConf - Cache configuration for hfile's contents

primaryReplicaReader - true if this is a reader for primary replica

conf - Configuration

Returns:

A version specific Hfile Reader

Throws:

IOException - If file is invalid, will throw CorruptHFileException flavored IOException

createReader

public static HFile.Reader createReader(org.apache.hadoop.fs.FileSystem fs,
                                        org.apache.hadoop.fs.Path path,
                                        org.apache.hadoop.conf.Configuration conf)
                                 throws IOException

Creates reader with cache configuration disabled

Parameters:

fs - filesystem

path - Path to file to read

Returns:

an active Reader instance

Throws:

IOException - Will throw a CorruptHFileException (DoNotRetryIOException subtype) if hfile is corrupt/invalid.

createReader

public static HFile.Reader createReader(org.apache.hadoop.fs.FileSystem fs,
                                        org.apache.hadoop.fs.Path path,
                                        CacheConfig cacheConf,
                                        boolean primaryReplicaReader,
                                        org.apache.hadoop.conf.Configuration conf)
                                 throws IOException

Parameters:

fs - filesystem

path - Path to file to read

cacheConf - This must not be null. @see CacheConfig.CacheConfig(Configuration)

primaryReplicaReader - true if this is a reader for primary replica

Returns:

an active Reader instance

Throws:

IOException - Will throw a CorruptHFileException (DoNotRetryIOException subtype) if hfile is corrupt/invalid.

createReaderFromStream

static HFile.Reader createReaderFromStream(org.apache.hadoop.fs.Path path,
                                           org.apache.hadoop.fs.FSDataInputStream fsdis,
                                           long size,
                                           CacheConfig cacheConf,
                                           org.apache.hadoop.conf.Configuration conf)
                                    throws IOException

This factory method is used only by unit tests.
The sockets and the file descriptors held by the method parameter FSDataInputStreamWrapper passed will be freed after its usage so caller needs to ensure that no other threads have access to the same passed reference.

Throws:

IOException

isHFileFormat

public static boolean isHFileFormat(org.apache.hadoop.fs.FileSystem fs,
                                    org.apache.hadoop.fs.Path path)
                             throws IOException

Returns true if the specified file has a valid HFile Trailer.

Parameters:

fs - filesystem

path - Path to file to verify

Returns:

true if the file has a valid HFile Trailer, otherwise false

Throws:

IOException - if failed to read from the underlying stream

isHFileFormat

public static boolean isHFileFormat(org.apache.hadoop.fs.FileSystem fs,
                                    org.apache.hadoop.fs.FileStatus fileStatus)
                             throws IOException

Returns true if the specified file has a valid HFile Trailer.

Parameters:

fs - filesystem

fileStatus - the file to verify

Returns:

true if the file has a valid HFile Trailer, otherwise false

Throws:

IOException - if failed to read from the underlying stream

isReservedFileInfoKey

public static boolean isReservedFileInfoKey(byte[] key)

Return true if the given file info key is reserved for internal use.

getSupportedCompressionAlgorithms

public static String[] getSupportedCompressionAlgorithms()

Get names of supported compression algorithms. The names are acceptable by HFile.Writer.

Returns:

Array of strings, each represents a supported compression algorithm. Currently, the following compression algorithms are supported.

• "none" - No compression.
• "gz" - GZIP compression.

longToInt

static int longToInt(long l)

getStoreFiles

static List<org.apache.hadoop.fs.Path> getStoreFiles(org.apache.hadoop.fs.FileSystem fs,
                                                     org.apache.hadoop.fs.Path regionDir)
                                              throws IOException

Returns all HFiles belonging to the given region directory. Could return an empty list.

Parameters:

fs - The file system reference.

regionDir - The region directory to scan.

Returns:

The list of files found.

Throws:

IOException - When scanning the files fails.

checkFormatVersion

public static void checkFormatVersion(int version)
                               throws IllegalArgumentException

Checks the given HFile format version, and throws an exception if invalid. Note that if the version number comes from an input file and has not been verified, the caller needs to re-throw an IOException to indicate that this is not a software error, but corrupted input.

Parameters:

version - an HFile version

Throws:

IllegalArgumentException - if the version is invalid

checkHFileVersion

public static void checkHFileVersion(org.apache.hadoop.conf.Configuration c)

main

public static void main(String[] args)
                 throws Exception

Throws:

Exception