Package org.apache.hadoop.hbase.io.hfile

Class FixedFileTrailer

java.lang.Object

org.apache.hadoop.hbase.io.hfile.FixedFileTrailer

@Private
public class FixedFileTrailer
extends Object

The HFile has a fixed trailer which contains offsets to other variable parts of the file. Also includes basic metadata on this file. The trailer size is fixed within a given HFile format version only, but we always store the version number as the last four-byte integer of the file. The version number itself is split into two portions, a major version and a minor version. The last three bytes of a file are the major version and a single preceding byte is the minor number. The major version determines which readers/writers to use to read/write a hfile while a minor version determines smaller changes in hfile format that do not need a new reader/writer type.

Field Summary

Fields

Modifier and Type	Field	Description
private String	comparatorClassName	Raw key comparator class name in version 3
private Compression.Algorithm	compressionCodec	The compression codec used for all blocks.
private int	dataIndexCount	The number of entries in the root data index.
private byte[]	encryptionKey	The encryption key
private long	entryCount	The number of key/value pairs in the file.
private long	fileInfoOffset	Offset to the fileinfo data, a small block of vitals.
private long	firstDataBlockOffset	The offset of the first data block.
private long	lastDataBlockOffset	It is guaranteed that no key/value data blocks start after this offset in the file.
private long	loadOnOpenDataOffset	In version 1, the offset to the data block index.
private static final org.slf4j.Logger	LOG
private final int	majorVersion	The HFile format major version.
private static final int	MAX_COMPARATOR_NAME_LENGTH	We store the comparator class name as a fixed-length field in the trailer.
private static final int	MAX_TRAILER_SIZE
private int	metaIndexCount	The number of entries in the meta index
private final int	minorVersion	The HFile format minor version.
private static final int	NOT_PB_SIZE
private int	numDataIndexLevels	The number of levels in the potentially multi-level data index.
private long	totalUncompressedBytes	The total uncompressed size of keys/values stored in the file.
private static final int[]	TRAILER_SIZE
private long	uncompressedDataIndexSize	Total uncompressed size of all blocks of the data index

Constructor Summary

Constructors

Constructor	Description
FixedFileTrailer(int majorVersion, int minorVersion)

Method Summary

Modifier and Type	Method	Description
private void	append(StringBuilder sb, String s)
private static int[]	computeTrailerSizeByVersion()
(package private) CellComparator	createComparator()
(package private) static CellComparator	createComparator(String comparatorClassName)
(package private) void	deserialize(DataInputStream inputStream)	Deserialize the fixed file trailer from the given stream.
(package private) void	deserializeFromPB(DataInputStream inputStream)	Deserialize the file trailer as protobuf
(package private) void	deserializeFromWritable(DataInput input)	Deserialize the file trailer as writable data
void	expectAtLeastMajorVersion(int lowerBound)
void	expectMajorVersion(int expected)
void	expectMinorVersion(int expected)
private static int	extractMajorVersion(int serializedVersion)	Extracts the major version for a 4-byte serialized version data.
private static int	extractMinorVersion(int serializedVersion)	Extracts the minor version for a 4-byte serialized version data.
private static Class<? extends CellComparator>	getComparatorClass(String comparatorClassName)
String	getComparatorClassName()
Compression.Algorithm	getCompressionCodec()
int	getDataIndexCount()
byte[]	getEncryptionKey()
long	getEntryCount()
long	getFileInfoOffset()
long	getFirstDataBlockOffset()
private String	getHBase1CompatibleName(String comparator)	Deprecated. Since hbase-2.0.0.
long	getLastDataBlockOffset()
long	getLoadOnOpenDataOffset()
int	getMajorVersion()	Returns the major version of this HFile format
private static int	getMaxTrailerSize()
int	getMetaIndexCount()
int	getMinorVersion()	Returns the minor version of this HFile format
int	getNumDataIndexLevels()
long	getTotalUncompressedBytes()
int	getTrailerSize()
(package private) static int	getTrailerSize(int version)
long	getUncompressedDataIndexSize()
(package private) static int	materializeVersion(int majorVersion, int minorVersion)	Create a 4 byte serialized version number by combining the minor and major version numbers.
static FixedFileTrailer	readFromStream(org.apache.hadoop.fs.FSDataInputStream istream, long fileSize)	Reads a file trailer from the given file.
(package private) void	serialize(DataOutputStream outputStream)	Write the trailer to a data stream.
(package private) void	serializeAsPB(DataOutputStream output)	Write trailer data as protobuf.
void	setComparatorClass(Class<? extends CellComparator> klass)
void	setCompressionCodec(Compression.Algorithm compressionCodec)
void	setDataIndexCount(int dataIndexCount)
void	setEncryptionKey(byte[] keyBytes)
void	setEntryCount(long newEntryCount)
void	setFileInfoOffset(long fileInfoOffset)
void	setFirstDataBlockOffset(long firstDataBlockOffset)
void	setLastDataBlockOffset(long lastDataBlockOffset)
void	setLoadOnOpenOffset(long loadOnOpenDataOffset)
void	setMetaIndexCount(int metaIndexCount)
void	setNumDataIndexLevels(int numDataIndexLevels)
void	setTotalUncompressedBytes(long totalUncompressedBytes)
void	setUncompressedDataIndexSize(long uncompressedDataIndexSize)
(package private) org.apache.hadoop.hbase.shaded.protobuf.generated.HFileProtos.FileTrailerProto	toProtobuf()
String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Details

LOG

private static final org.slf4j.Logger LOG

MAX_COMPARATOR_NAME_LENGTH

private static final int MAX_COMPARATOR_NAME_LENGTH

We store the comparator class name as a fixed-length field in the trailer.

See Also:

• Constant Field Values

fileInfoOffset

private long fileInfoOffset

Offset to the fileinfo data, a small block of vitals. Necessary in v1 but only potentially useful for pretty-printing in v2.

loadOnOpenDataOffset

private long loadOnOpenDataOffset

In version 1, the offset to the data block index. Starting from version 2, the meaning of this field is the offset to the section of the file that should be loaded at the time the file is being opened: i.e. on open we load the root index, file info, etc. See http://hbase.apache.org/book.html#_hfile_format_2 in the reference guide.

dataIndexCount

private int dataIndexCount

The number of entries in the root data index.

uncompressedDataIndexSize

private long uncompressedDataIndexSize

Total uncompressed size of all blocks of the data index

metaIndexCount

private int metaIndexCount

The number of entries in the meta index

totalUncompressedBytes

private long totalUncompressedBytes

The total uncompressed size of keys/values stored in the file.

entryCount

private long entryCount

The number of key/value pairs in the file. This field was int in version 1, but is now long.

compressionCodec

private Compression.Algorithm compressionCodec

The compression codec used for all blocks.

numDataIndexLevels

private int numDataIndexLevels

The number of levels in the potentially multi-level data index. Used from version 2 onwards.

firstDataBlockOffset

private long firstDataBlockOffset

The offset of the first data block.

lastDataBlockOffset

private long lastDataBlockOffset

It is guaranteed that no key/value data blocks start after this offset in the file.

comparatorClassName

private String comparatorClassName

Raw key comparator class name in version 3

encryptionKey

private byte[] encryptionKey

The encryption key

majorVersion

private final int majorVersion

The HFile format major version.

minorVersion

private final int minorVersion

The HFile format minor version.

TRAILER_SIZE

private static final int[] TRAILER_SIZE

MAX_TRAILER_SIZE

private static final int MAX_TRAILER_SIZE

NOT_PB_SIZE

private static final int NOT_PB_SIZE

See Also:

• Constant Field Values

Constructor Details

FixedFileTrailer

FixedFileTrailer(int majorVersion,
 int minorVersion)

Method Details

computeTrailerSizeByVersion

private static int[] computeTrailerSizeByVersion()

getMaxTrailerSize

private static int getMaxTrailerSize()

getTrailerSize

static int getTrailerSize(int version)

getTrailerSize

public int getTrailerSize()

serialize

void serialize(DataOutputStream outputStream)
        throws IOException

Write the trailer to a data stream. We support writing version 1 for testing and for determining version 1 trailer size. It is also easy to see what fields changed in version 2.

Throws:

IOException

toProtobuf

org.apache.hadoop.hbase.shaded.protobuf.generated.HFileProtos.FileTrailerProto toProtobuf()

serializeAsPB

void serializeAsPB(DataOutputStream output)
            throws IOException

Write trailer data as protobuf. NOTE: we run a translation on the comparator name and will serialize the old hbase-1.x where it makes sense. See getHBase1CompatibleName(String).

Throws:

IOException

deserialize

void deserialize(DataInputStream inputStream)
          throws IOException

Deserialize the fixed file trailer from the given stream. The version needs to already be specified. Make sure this is consistent with serialize(DataOutputStream).

Throws:

IOException

deserializeFromPB

void deserializeFromPB(DataInputStream inputStream)
                throws IOException

Deserialize the file trailer as protobuf

Throws:

IOException

deserializeFromWritable

void deserializeFromWritable(DataInput input)
                      throws IOException

Deserialize the file trailer as writable data

Throws:

IOException

append

private void append(StringBuilder sb,
 String s)

toString

public String toString()

Overrides:

toString in class Object

readFromStream

public static FixedFileTrailer readFromStream(org.apache.hadoop.fs.FSDataInputStream istream,
 long fileSize)
                                       throws IOException

Reads a file trailer from the given file.

Parameters:

istream - the input stream with the ability to seek. Does not have to be buffered, as only one read operation is made.

fileSize - the file size. Can be obtained using FileSystem.getFileStatus(org.apache.hadoop.fs.Path).

Returns:

the fixed file trailer read

Throws:

IOException - if failed to read from the underlying stream, or the trailer is corrupted, or the version of the trailer is unsupported

expectMajorVersion

public void expectMajorVersion(int expected)

expectMinorVersion

public void expectMinorVersion(int expected)

expectAtLeastMajorVersion

public void expectAtLeastMajorVersion(int lowerBound)

getFileInfoOffset

public long getFileInfoOffset()

setFileInfoOffset

public void setFileInfoOffset(long fileInfoOffset)

getLoadOnOpenDataOffset

public long getLoadOnOpenDataOffset()

setLoadOnOpenOffset

public void setLoadOnOpenOffset(long loadOnOpenDataOffset)

getDataIndexCount

public int getDataIndexCount()

setDataIndexCount

public void setDataIndexCount(int dataIndexCount)

getMetaIndexCount

public int getMetaIndexCount()

setMetaIndexCount

public void setMetaIndexCount(int metaIndexCount)

getTotalUncompressedBytes

public long getTotalUncompressedBytes()

setTotalUncompressedBytes

public void setTotalUncompressedBytes(long totalUncompressedBytes)

getEntryCount

public long getEntryCount()

setEntryCount

public void setEntryCount(long newEntryCount)

getCompressionCodec

public Compression.Algorithm getCompressionCodec()

setCompressionCodec

public void setCompressionCodec(Compression.Algorithm compressionCodec)

getNumDataIndexLevels

public int getNumDataIndexLevels()

setNumDataIndexLevels

public void setNumDataIndexLevels(int numDataIndexLevels)

getLastDataBlockOffset

public long getLastDataBlockOffset()

setLastDataBlockOffset

public void setLastDataBlockOffset(long lastDataBlockOffset)

getFirstDataBlockOffset

public long getFirstDataBlockOffset()

setFirstDataBlockOffset

public void setFirstDataBlockOffset(long firstDataBlockOffset)

getComparatorClassName

public String getComparatorClassName()

getMajorVersion

public int getMajorVersion()

Returns the major version of this HFile format

getMinorVersion

public int getMinorVersion()

Returns the minor version of this HFile format

setComparatorClass

public void setComparatorClass(Class<? extends CellComparator> klass)

getHBase1CompatibleName

@Deprecated
private String getHBase1CompatibleName(String comparator)

Deprecated.

Since hbase-2.0.0. Will be removed in hbase-3.0.0.

If a 'standard' Comparator, write the old name for the Comparator when we serialize rather than the new name; writing the new name will make it so newly-written hfiles are not parseable by hbase-1.x, a facility we'd like to preserve across rolling upgrade and hbase-1.x clusters reading hbase-2.x produce.

The Comparators in hbase-2.x work the same as they did in hbase-1.x; they compare KeyValues. In hbase-2.x they were renamed making use of the more generic 'Cell' nomenclature to indicate that we intend to move away from KeyValues post hbase-2. A naming change is not reason enough to make it so hbase-1.x cannot read hbase-2.x files given the structure goes unchanged (hfile v3). So, lets write the old names for Comparators into the hfile tails in hbase-2. Here is where we do the translation. getComparatorClass(String) does translation going the other way.

The translation is done on the serialized Protobuf only.

Parameters:

comparator - String class name of the Comparator used in this hfile.

Returns:

What to store in the trailer as our comparator name.

Since:

hbase-2.0.0.

See Also:

• getComparatorClass(String)

getComparatorClass

private static Class<? extends CellComparator> getComparatorClass(String comparatorClassName)
                                                           throws IOException

Throws:

IOException

createComparator

static CellComparator createComparator(String comparatorClassName)
                                throws IOException

Throws:

IOException

createComparator

CellComparator createComparator()
                         throws IOException

Throws:

IOException

getUncompressedDataIndexSize

public long getUncompressedDataIndexSize()

setUncompressedDataIndexSize

public void setUncompressedDataIndexSize(long uncompressedDataIndexSize)

getEncryptionKey

public byte[] getEncryptionKey()

setEncryptionKey

public void setEncryptionKey(byte[] keyBytes)

extractMajorVersion

private static int extractMajorVersion(int serializedVersion)

Extracts the major version for a 4-byte serialized version data. The major version is the 3 least significant bytes

extractMinorVersion

private static int extractMinorVersion(int serializedVersion)

Extracts the minor version for a 4-byte serialized version data. The major version are the 3 the most significant bytes

materializeVersion

static int materializeVersion(int majorVersion,
 int minorVersion)

Create a 4 byte serialized version number by combining the minor and major version numbers.