org.apache.hadoop.hbase.io.hfile

Class HFileInfo

java.lang.Object

org.apache.hadoop.hbase.io.hfile.HFileInfo

All Implemented Interfaces:

Map<byte[],byte[]>, SortedMap<byte[],byte[]>

@InterfaceAudience.Private
public class HFileInfo
extends Object
implements SortedMap<byte[],byte[]>

Metadata Map of attributes for HFile written out as HFile Trailer. Created by the Writer and added to the tail of the file just before close. Metadata includes core attributes such as last key seen, comparator used writing the file, etc. Clients can add their own attributes via append(byte[], byte[], boolean) and they'll be persisted and available at read time. Reader creates the HFileInfo on open by reading the tail of the HFile. The parse of the HFile trailer also creates a HFileContext, a read-only data structure that includes bulk of the HFileInfo and extras that is safe to pass around when working on HFiles.

See Also:

HFileContext

Nested Class Summary

Nested classes/interfaces inherited from interface java.util.Map

Map.Entry<K,V>

Field Summary

Fields

Modifier and Type	Field and Description
(package private) static byte[]	AVG_KEY_LEN
(package private) static byte[]	AVG_VALUE_LEN
private int	avgKeyLen Average key length read from file info
private int	avgValueLen Average value length read from file info
private HFileBlock.BlockIterator	blockIter The iterator will track all blocks in load-on-open section, since we use the ByteBuffAllocator to manage the ByteBuffers in block now, so we must ensure that deallocate all ByteBuffers in the end.
(package private) static byte[]	CREATE_TIME_TS
private HFileBlockIndex.CellBasedKeyBlockIndexReader	dataIndexReader
private boolean	decodeMemstoreTS
private HFileContext	hfileContext
private boolean	includesMemstoreTS
(package private) static byte[]	LASTKEY
private Cell	lastKeyCell Last key in the file.
private List<HFileBlock>	loadOnOpenBlocks Blocks read from the load-on-open section, excluding data root index, meta index, and file info.
private static org.slf4j.Logger	LOG
private SortedMap<byte[],byte[]>	map
(package private) static int	MAX_MINOR_VERSION Maximum minor version supported by this HFile format
static byte[]	MAX_TAGS_LEN
private HFileBlockIndex.ByteArrayKeyBlockIndexReader	metaIndexReader
private static int	MIN_V2_MINOR_VERSION_WITH_PB We can read files whose major version is v2 IFF their minor version is at least 3.
(package private) static String	RESERVED_PREFIX
(package private) static byte[]	RESERVED_PREFIX_BYTES
(package private) static byte[]	TAGS_COMPRESSED
private FixedFileTrailer	trailer

Constructor Summary

Constructors

Constructor and Description
HFileInfo()
HFileInfo(ReaderContext context, org.apache.hadoop.conf.Configuration conf)

Method Summary

Modifier and Type	Method and Description
HFileInfo	append(byte[] k, byte[] v, boolean checkPrefix) Append the given key/value pair to the file info, optionally checking the key prefix.
private void	checkFileVersion(org.apache.hadoop.fs.Path path) File version check is a little sloppy.
void	clear()
void	close()
Comparator<? super byte[]>	comparator()
boolean	containsKey(Object key)
boolean	containsValue(Object value)
private HFileContext	createHFileContext(org.apache.hadoop.fs.Path path, FixedFileTrailer trailer, org.apache.hadoop.conf.Configuration conf)
Set<Map.Entry<byte[],byte[]>>	entrySet()
boolean	equals(Object o)
byte[]	firstKey()
byte[]	get(Object key)
int	getAvgKeyLen()
int	getAvgValueLen()
HFileBlockIndex.CellBasedKeyBlockIndexReader	getDataBlockIndexReader()
HFileContext	getHFileContext()
Cell	getLastKeyCell()
List<HFileBlock>	getLoadOnOpenBlocks()
int	getMajorVersion()
HFileBlockIndex.ByteArrayKeyBlockIndexReader	getMetaBlockIndexReader()
FixedFileTrailer	getTrailer()
int	hashCode()
SortedMap<byte[],byte[]>	headMap(byte[] toKey)
void	initMetaAndIndex(HFile.Reader reader) should be called after initTrailerAndContext
void	initTrailerAndContext(ReaderContext context, org.apache.hadoop.conf.Configuration conf)
boolean	isDecodeMemstoreTS()
boolean	isEmpty()
static boolean	isReservedFileInfoKey(byte[] key) Return true if the given file info key is reserved for internal use.
Set<byte[]>	keySet()
byte[]	lastKey()
private void	loadMetaInfo(HFileBlock.BlockIterator blockIter, HFileContext hfileContext)
(package private) void	parsePB(org.apache.hadoop.hbase.shaded.protobuf.generated.HFileProtos.FileInfoProto fip) Fill our map with content of the pb we read off disk
(package private) void	parseWritable(DataInputStream in) Now parse the old Writable format.
byte[]	put(byte[] key, byte[] value)
void	putAll(Map<? extends byte[],? extends byte[]> m)
(package private) void	read(DataInputStream in) Populate this instance with what we find on the passed in in stream.
byte[]	remove(Object key)
void	setTrailer(FixedFileTrailer trailer)
boolean	shouldIncludeMemStoreTS()
int	size()
SortedMap<byte[],byte[]>	subMap(byte[] fromKey, byte[] toKey)
SortedMap<byte[],byte[]>	tailMap(byte[] fromKey)
Collection<byte[]>	values()
(package private) void	write(DataOutputStream out) Write out this instance on the passed in out stream.

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.util.Map

compute, computeIfAbsent, computeIfPresent, forEach, getOrDefault, merge, putIfAbsent, remove, replace, replace, replaceAll

Field Detail

LOG

private static final org.slf4j.Logger LOG

RESERVED_PREFIX

static final String RESERVED_PREFIX

See Also:

Constant Field Values

RESERVED_PREFIX_BYTES

static final byte[] RESERVED_PREFIX_BYTES

LASTKEY

static final byte[] LASTKEY

AVG_KEY_LEN

static final byte[] AVG_KEY_LEN

AVG_VALUE_LEN

static final byte[] AVG_VALUE_LEN

CREATE_TIME_TS

static final byte[] CREATE_TIME_TS

TAGS_COMPRESSED

static final byte[] TAGS_COMPRESSED

MAX_TAGS_LEN

public static final byte[] MAX_TAGS_LEN

map

private final SortedMap<byte[],byte[]> map

MIN_V2_MINOR_VERSION_WITH_PB

private static final int MIN_V2_MINOR_VERSION_WITH_PB

We can read files whose major version is v2 IFF their minor version is at least 3.

See Also:

Constant Field Values

MAX_MINOR_VERSION

static final int MAX_MINOR_VERSION

Maximum minor version supported by this HFile format

See Also:

Constant Field Values

lastKeyCell

private Cell lastKeyCell

Last key in the file. Filled in when we read in the file info

avgKeyLen

private int avgKeyLen

Average key length read from file info

avgValueLen

private int avgValueLen

Average value length read from file info

includesMemstoreTS

private boolean includesMemstoreTS

decodeMemstoreTS

private boolean decodeMemstoreTS

loadOnOpenBlocks

private List<HFileBlock> loadOnOpenBlocks

Blocks read from the load-on-open section, excluding data root index, meta index, and file info.

blockIter

private HFileBlock.BlockIterator blockIter

The iterator will track all blocks in load-on-open section, since we use the ByteBuffAllocator to manage the ByteBuffers in block now, so we must ensure that deallocate all ByteBuffers in the end.

dataIndexReader

private HFileBlockIndex.CellBasedKeyBlockIndexReader dataIndexReader

metaIndexReader

private HFileBlockIndex.ByteArrayKeyBlockIndexReader metaIndexReader

trailer

private FixedFileTrailer trailer

hfileContext

private HFileContext hfileContext

Constructor Detail

HFileInfo

public HFileInfo()

HFileInfo

public HFileInfo(ReaderContext context,
                 org.apache.hadoop.conf.Configuration conf)
          throws IOException

Throws:

IOException

Method Detail

append

public HFileInfo append(byte[] k,
                        byte[] v,
                        boolean checkPrefix)
                 throws IOException

Append the given key/value pair to the file info, optionally checking the key prefix.

Parameters:

k - key to add

v - value to add

checkPrefix - whether to check that the provided key does not start with the reserved prefix

Returns:

this file info object

Throws:

IOException - if the key or value is invalid

isReservedFileInfoKey

public static boolean isReservedFileInfoKey(byte[] key)

Return true if the given file info key is reserved for internal use.

clear

public void clear()

Specified by:

clear in interface Map<byte[],byte[]>

comparator

public Comparator<? super byte[]> comparator()

Specified by:

comparator in interface SortedMap<byte[],byte[]>

containsKey

public boolean containsKey(Object key)

Specified by:

containsKey in interface Map<byte[],byte[]>

containsValue

public boolean containsValue(Object value)

Specified by:

containsValue in interface Map<byte[],byte[]>

entrySet

public Set<Map.Entry<byte[],byte[]>> entrySet()

Specified by:

entrySet in interface Map<byte[],byte[]>

Specified by:

entrySet in interface SortedMap<byte[],byte[]>

equals

public boolean equals(Object o)

Specified by:

equals in interface Map<byte[],byte[]>

Overrides:

equals in class Object

firstKey

public byte[] firstKey()

Specified by:

firstKey in interface SortedMap<byte[],byte[]>

get

public byte[] get(Object key)

Specified by:

get in interface Map<byte[],byte[]>

hashCode

public int hashCode()

Specified by:

hashCode in interface Map<byte[],byte[]>

Overrides:

hashCode in class Object

headMap

public SortedMap<byte[],byte[]> headMap(byte[] toKey)

Specified by:

headMap in interface SortedMap<byte[],byte[]>

isEmpty

public boolean isEmpty()

Specified by:

isEmpty in interface Map<byte[],byte[]>

keySet

public Set<byte[]> keySet()

Specified by:

keySet in interface Map<byte[],byte[]>

Specified by:

keySet in interface SortedMap<byte[],byte[]>

lastKey

public byte[] lastKey()

Specified by:

lastKey in interface SortedMap<byte[],byte[]>

put

public byte[] put(byte[] key,
                  byte[] value)

Specified by:

put in interface Map<byte[],byte[]>

putAll

public void putAll(Map<? extends byte[],? extends byte[]> m)

Specified by:

putAll in interface Map<byte[],byte[]>

remove

public byte[] remove(Object key)

Specified by:

remove in interface Map<byte[],byte[]>

size

public int size()

Specified by:

size in interface Map<byte[],byte[]>

subMap

public SortedMap<byte[],byte[]> subMap(byte[] fromKey,
                                       byte[] toKey)

Specified by:

subMap in interface SortedMap<byte[],byte[]>

tailMap

public SortedMap<byte[],byte[]> tailMap(byte[] fromKey)

Specified by:

tailMap in interface SortedMap<byte[],byte[]>

values

public Collection<byte[]> values()

Specified by:

values in interface Map<byte[],byte[]>

Specified by:

values in interface SortedMap<byte[],byte[]>

write

void write(DataOutputStream out)
    throws IOException

Write out this instance on the passed in out stream. We write it as a protobuf.

Throws:

IOException

See Also:

read(DataInputStream)

read

void read(DataInputStream in)
   throws IOException

Populate this instance with what we find on the passed in in stream. Can deserialize protobuf of old Writables format.

Throws:

IOException

See Also:

write(DataOutputStream)

parseWritable

void parseWritable(DataInputStream in)
            throws IOException

Now parse the old Writable format. It was a list of Map entries. Each map entry was a key and a value of a byte []. The old map format had a byte before each entry that held a code which was short for the key or value type. We know it was a byte [] so in below we just read and dump it.

Throws:

IOException

parsePB

void parsePB(org.apache.hadoop.hbase.shaded.protobuf.generated.HFileProtos.FileInfoProto fip)

Fill our map with content of the pb we read off disk

Parameters:

fip - protobuf message to read

initTrailerAndContext

public void initTrailerAndContext(ReaderContext context,
                                  org.apache.hadoop.conf.Configuration conf)
                           throws IOException

Throws:

IOException

initMetaAndIndex

public void initMetaAndIndex(HFile.Reader reader)
                      throws IOException

should be called after initTrailerAndContext

Throws:

IOException

createHFileContext

private HFileContext createHFileContext(org.apache.hadoop.fs.Path path,
                                        FixedFileTrailer trailer,
                                        org.apache.hadoop.conf.Configuration conf)
                                 throws IOException

Throws:

IOException

loadMetaInfo

private void loadMetaInfo(HFileBlock.BlockIterator blockIter,
                          HFileContext hfileContext)
                   throws IOException

Throws:

IOException

checkFileVersion

private void checkFileVersion(org.apache.hadoop.fs.Path path)

File version check is a little sloppy. We read v3 files but can also read v2 files if their content has been pb'd; files written with 0.98.

close

public void close()

getMajorVersion

public int getMajorVersion()

setTrailer

public void setTrailer(FixedFileTrailer trailer)

getTrailer

public FixedFileTrailer getTrailer()

getDataBlockIndexReader

public HFileBlockIndex.CellBasedKeyBlockIndexReader getDataBlockIndexReader()

getMetaBlockIndexReader

public HFileBlockIndex.ByteArrayKeyBlockIndexReader getMetaBlockIndexReader()

getHFileContext

public HFileContext getHFileContext()

getLoadOnOpenBlocks

public List<HFileBlock> getLoadOnOpenBlocks()

getLastKeyCell

public Cell getLastKeyCell()

getAvgKeyLen

public int getAvgKeyLen()

getAvgValueLen

public int getAvgValueLen()

shouldIncludeMemStoreTS

public boolean shouldIncludeMemStoreTS()

isDecodeMemstoreTS

public boolean isDecodeMemstoreTS()