Package org.apache.hadoop.hbase.io.hfile
Class FixedFileTrailer
java.lang.Object
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer
The
HFile
has a fixed trailer which contains offsets to other variable parts of the file.
Also includes basic metadata on this file. The trailer size is fixed within a given HFile
format version only, but we always store the version number as the last four-byte integer of the
file. The version number itself is split into two portions, a major version and a minor version.
The last three bytes of a file are the major version and a single preceding byte is the minor
number. The major version determines which readers/writers to use to read/write a hfile while a
minor version determines smaller changes in hfile format that do not need a new reader/writer
type.-
Field Summary
Modifier and TypeFieldDescriptionprivate String
Raw key comparator class name in version 3private Compression.Algorithm
The compression codec used for all blocks.private int
The number of entries in the root data index.private byte[]
The encryption keyprivate long
The number of key/value pairs in the file.private long
Offset to the fileinfo data, a small block of vitals.private long
The offset of the first data block.private long
It is guaranteed that no key/value data blocks start after this offset in the file.private long
In version 1, the offset to the data block index.private static final org.slf4j.Logger
private final int
TheHFile
format major version.private static final int
We store the comparator class name as a fixed-length field in the trailer.private static final int
private int
The number of entries in the meta indexprivate final int
TheHFile
format minor version.private static final int
private int
The number of levels in the potentially multi-level data index.private long
The total uncompressed size of keys/values stored in the file.private static final int[]
private long
Total uncompressed size of all blocks of the data index -
Constructor Summary
-
Method Summary
Modifier and TypeMethodDescriptionprivate void
append
(StringBuilder sb, String s) private static int[]
(package private) CellComparator
(package private) static CellComparator
createComparator
(String comparatorClassName) (package private) void
deserialize
(DataInputStream inputStream) Deserialize the fixed file trailer from the given stream.(package private) void
deserializeFromPB
(DataInputStream inputStream) Deserialize the file trailer as protobuf(package private) void
deserializeFromWritable
(DataInput input) Deserialize the file trailer as writable datavoid
expectAtLeastMajorVersion
(int lowerBound) void
expectMajorVersion
(int expected) void
expectMinorVersion
(int expected) private static int
extractMajorVersion
(int serializedVersion) Extracts the major version for a 4-byte serialized version data.private static int
extractMinorVersion
(int serializedVersion) Extracts the minor version for a 4-byte serialized version data.private static Class<? extends CellComparator>
getComparatorClass
(String comparatorClassName) int
byte[]
long
long
long
long
long
int
Returns the major version of this HFile formatprivate static int
int
int
Returns the minor version of this HFile formatint
long
int
(package private) static int
getTrailerSize
(int version) long
(package private) static int
materializeVersion
(int majorVersion, int minorVersion) Create a 4 byte serialized version number by combining the minor and major version numbers.static FixedFileTrailer
readFromStream
(org.apache.hadoop.fs.FSDataInputStream istream, long fileSize) Reads a file trailer from the given file.(package private) void
serialize
(DataOutputStream outputStream) Write the trailer to a data stream.(package private) void
serializeAsPB
(DataOutputStream output) Write trailer data as protobuf.void
setComparatorClass
(Class<? extends CellComparator> klass) void
setCompressionCodec
(Compression.Algorithm compressionCodec) void
setDataIndexCount
(int dataIndexCount) void
setEncryptionKey
(byte[] keyBytes) void
setEntryCount
(long newEntryCount) void
setFileInfoOffset
(long fileInfoOffset) void
setFirstDataBlockOffset
(long firstDataBlockOffset) void
setLastDataBlockOffset
(long lastDataBlockOffset) void
setLoadOnOpenOffset
(long loadOnOpenDataOffset) void
setMetaIndexCount
(int metaIndexCount) void
setNumDataIndexLevels
(int numDataIndexLevels) void
setTotalUncompressedBytes
(long totalUncompressedBytes) void
setUncompressedDataIndexSize
(long uncompressedDataIndexSize) (package private) org.apache.hadoop.hbase.shaded.protobuf.generated.HFileProtos.FileTrailerProto
toString()
-
Field Details
-
LOG
-
MAX_COMPARATOR_NAME_LENGTH
We store the comparator class name as a fixed-length field in the trailer.- See Also:
-
fileInfoOffset
Offset to the fileinfo data, a small block of vitals. Necessary in v1 but only potentially useful for pretty-printing in v2. -
loadOnOpenDataOffset
In version 1, the offset to the data block index. Starting from version 2, the meaning of this field is the offset to the section of the file that should be loaded at the time the file is being opened: i.e. on open we load the root index, file info, etc. See http://hbase.apache.org/book.html#_hfile_format_2 in the reference guide. -
dataIndexCount
The number of entries in the root data index. -
uncompressedDataIndexSize
Total uncompressed size of all blocks of the data index -
metaIndexCount
The number of entries in the meta index -
totalUncompressedBytes
The total uncompressed size of keys/values stored in the file. -
entryCount
The number of key/value pairs in the file. This field was int in version 1, but is now long. -
compressionCodec
The compression codec used for all blocks. -
numDataIndexLevels
The number of levels in the potentially multi-level data index. Used from version 2 onwards. -
firstDataBlockOffset
The offset of the first data block. -
lastDataBlockOffset
It is guaranteed that no key/value data blocks start after this offset in the file. -
comparatorClassName
Raw key comparator class name in version 3 -
encryptionKey
The encryption key -
majorVersion
TheHFile
format major version. -
minorVersion
TheHFile
format minor version. -
TRAILER_SIZE
-
MAX_TRAILER_SIZE
-
NOT_PB_SIZE
- See Also:
-
-
Constructor Details
-
FixedFileTrailer
FixedFileTrailer(int majorVersion, int minorVersion)
-
-
Method Details
-
computeTrailerSizeByVersion
-
getMaxTrailerSize
-
getTrailerSize
-
getTrailerSize
-
serialize
Write the trailer to a data stream. We support writing version 1 for testing and for determining version 1 trailer size. It is also easy to see what fields changed in version 2.- Throws:
IOException
-
toProtobuf
org.apache.hadoop.hbase.shaded.protobuf.generated.HFileProtos.FileTrailerProto toProtobuf() -
serializeAsPB
Write trailer data as protobuf.- Throws:
IOException
-
deserialize
Deserialize the fixed file trailer from the given stream. The version needs to already be specified. Make sure this is consistent withserialize(DataOutputStream)
.- Throws:
IOException
-
deserializeFromPB
Deserialize the file trailer as protobuf- Throws:
IOException
-
deserializeFromWritable
Deserialize the file trailer as writable data- Throws:
IOException
-
append
-
toString
-
readFromStream
public static FixedFileTrailer readFromStream(org.apache.hadoop.fs.FSDataInputStream istream, long fileSize) throws IOException Reads a file trailer from the given file.- Parameters:
istream
- the input stream with the ability to seek. Does not have to be buffered, as only one read operation is made.fileSize
- the file size. Can be obtained usingFileSystem.getFileStatus(org.apache.hadoop.fs.Path)
.- Returns:
- the fixed file trailer read
- Throws:
IOException
- if failed to read from the underlying stream, or the trailer is corrupted, or the version of the trailer is unsupported
-
expectMajorVersion
-
expectMinorVersion
-
expectAtLeastMajorVersion
-
getFileInfoOffset
-
setFileInfoOffset
-
getLoadOnOpenDataOffset
-
setLoadOnOpenOffset
-
getDataIndexCount
-
setDataIndexCount
-
getMetaIndexCount
-
setMetaIndexCount
-
getTotalUncompressedBytes
-
setTotalUncompressedBytes
-
getEntryCount
-
setEntryCount
-
getCompressionCodec
-
setCompressionCodec
-
getNumDataIndexLevels
-
setNumDataIndexLevels
-
getLastDataBlockOffset
-
setLastDataBlockOffset
-
getFirstDataBlockOffset
-
setFirstDataBlockOffset
-
getComparatorClassName
-
getMajorVersion
Returns the major version of this HFile format -
getMinorVersion
Returns the minor version of this HFile format -
setComparatorClass
-
getComparatorClass
private static Class<? extends CellComparator> getComparatorClass(String comparatorClassName) throws IOException - Throws:
IOException
-
createComparator
- Throws:
IOException
-
createComparator
- Throws:
IOException
-
getUncompressedDataIndexSize
-
setUncompressedDataIndexSize
-
getEncryptionKey
-
setEncryptionKey
-
extractMajorVersion
Extracts the major version for a 4-byte serialized version data. The major version is the 3 least significant bytes -
extractMinorVersion
Extracts the minor version for a 4-byte serialized version data. The major version are the 3 the most significant bytes -
materializeVersion
Create a 4 byte serialized version number by combining the minor and major version numbers.
-