Package org.apache.hadoop.hbase.regionserver

Class HStoreFile

java.lang.Object

org.apache.hadoop.hbase.regionserver.HStoreFile

All Implemented Interfaces:

StoreFile

@Private
public class HStoreFile
extends Object
implements StoreFile

A Store data file. Stores usually have one or more of these files. They are produced by flushing the memstore to disk. To create, instantiate a writer using StoreFileWriter.Builder and append data. Be sure to add any metadata before calling close on the Writer (Use the appendMetadata convenience methods). On close, a StoreFile is sitting in the Filesystem. To refer to it, create a StoreFile instance passing filesystem and path. To read, call initReader()

StoreFiles may also reference store files in another Store. The reason for this weird pattern where you use a different instance for the writer and a reader is that we write once but read a lot more.

Field Summary

Fields

Modifier and Type	Field	Description
static final byte[]	BLOOM_FILTER_PARAM_KEY	Bloom filter param in FileInfo
static final byte[]	BLOOM_FILTER_TYPE_KEY	Bloom filter Type in FileInfo
static final byte[]	BULKLOAD_TASK_KEY	Meta key set when store file is a result of a bulk load
static final byte[]	BULKLOAD_TIME_KEY
private final CacheConfig	cacheConf
private final BloomType	cfBloomType	Bloom filter type specified in column family configuration.
private boolean	compactedAway
private final Set<String>	compactedStoreFiles
static final byte[]	COMPACTION_EVENT_KEY	Key for compaction event which contains the compacted storefiles in FileInfo
private CellComparator	comparator
static final byte[]	DELETE_FAMILY_COUNT	Delete Family Count in FileInfo
static final byte[]	EARLIEST_PUT_TS	Key for timestamp of earliest-put in metadata
static final byte[]	EXCLUDE_FROM_MINOR_COMPACTION_KEY	Minor compaction flag in FileInfo
private boolean	excludeFromMinorCompaction
private final StoreFileInfo	fileInfo
private Optional<Cell>	firstKey
private StoreFileReader	initialReader
private InputStreamBlockDistribution	initialReaderBlockDistribution
static final byte[]	LAST_BLOOM_KEY	Last Bloom filter key in FileInfo
private Optional<Cell>	lastKey
private static final org.slf4j.Logger	LOG
static final byte[]	MAJOR_COMPACTION_KEY	Major compaction flag in FileInfo
private AtomicBoolean	majorCompaction
static final byte[]	MAX_SEQ_ID_KEY	Max Sequence ID in FileInfo
private long	maxMemstoreTS
private Map<byte[],byte[]>	metadataMap	Map of the metadata entries in the corresponding HFile.
private final BloomFilterMetrics	metrics
static final byte[]	MOB_CELLS_COUNT	Key for the number of mob cells in metadata
static final byte[]	MOB_FILE_REFS	Key for the list of MOB file references
static final byte[]	NULL_VALUE	Null data
private long	sequenceid
static final byte[]	SKIP_RESET_SEQ_ID	Key for skipping resetting sequence id in metadata.
static final byte[]	TIMERANGE_KEY	Key for Timerange information in metadata

Constructor Summary

Constructors

Constructor	Description
HStoreFile(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path p, org.apache.hadoop.conf.Configuration conf, CacheConfig cacheConf, BloomType cfBloomType, boolean primaryReplica)	Constructor, loads a reader and it's indices, etc.
HStoreFile(StoreFileInfo fileInfo, BloomType cfBloomType, CacheConfig cacheConf)	Constructor, loads a reader and it's indices, etc.
HStoreFile(StoreFileInfo fileInfo, BloomType cfBloomType, CacheConfig cacheConf, BloomFilterMetrics metrics)	Constructor, loads a reader and it's indices, etc.

Method Summary

Modifier and Type	Method	Description
void	closeStoreFile(boolean evictOnClose)
private StoreFileReader	createStreamReader(boolean canUseDropBehind)
(package private) long	decreaseRefCount()
(package private) static void	decreaseStoreFilesRefeCount(Collection<HStoreFile> storeFiles)
void	deleteStoreFile()	Delete this file
boolean	excludeFromMinorCompaction()	Returns True if this file should not be part of a minor compaction.
OptionalLong	getBulkLoadTimestamp()	Return the timestamp at which this bulk load file was generated.
CacheConfig	getCacheConf()
(package private) Set<String>	getCompactedStoreFiles()
CellComparator	getComparator()	Get the comparator for comparing two cells.
org.apache.hadoop.fs.Path	getEncodedPath()	Returns Encoded Path if this StoreFile was made with a Stream.
StoreFileInfo	getFileInfo()
Optional<Cell>	getFirstKey()	Get the first key in this store file.
HDFSBlocksDistribution	getHDFSBlockDistribution()
Optional<Cell>	getLastKey()	Get the last key in this store file.
OptionalLong	getMaximumTimestamp()	Get the max timestamp of all the cells in the store file.
long	getMaxMemStoreTS()	Get max of the MemstoreTS in the KV's in this store file.
long	getMaxSequenceId()	Returns This files maximum edit sequence id.
byte[]	getMetadataValue(byte[] key)
OptionalLong	getMinimumTimestamp()	Get the min timestamp of all the cells in the store file.
long	getModificationTimestamp()	Get the modification time of this store file.
long	getModificationTimeStamp()	Get the modification time of this store file.
org.apache.hadoop.fs.Path	getPath()	Returns Path or null if this StoreFile was made with a Stream.
StoreFileScanner	getPreadScanner(boolean cacheBlocks, long readPt, long scannerOrder, boolean canOptimizeForNonNullColumn)	Get a scanner which uses pread.
org.apache.hadoop.fs.Path	getQualifiedPath()	Returns Returns the qualified path of this StoreFile
StoreFileReader	getReader()
int	getRefCount()
StoreFileScanner	getStreamScanner(boolean canUseDropBehind, boolean cacheBlocks, boolean isCompaction, long readPt, long scannerOrder, boolean canOptimizeForNonNullColumn)	Get a scanner which uses streaming read.
(package private) long	increaseRefCount()
(package private) static void	increaseStoreFilesRefeCount(Collection<HStoreFile> storeFiles)
void	initReader()	Initialize the reader used for pread.
boolean	isBulkLoadResult()	Check if this storefile was created by bulk load.
boolean	isCompactedAway()
boolean	isHFile()	Returns True if this is HFile.
boolean	isMajorCompactionResult()	Returns True if this file was made by a major compaction.
boolean	isReference()	Returns True if this is a StoreFile Reference.
boolean	isReferencedInReads()	Returns true if the file is still used in reads
private boolean	isSkipResetSeqId(byte[] skipResetSeqId)	Gets whether to skip resetting the sequence id for cells.
void	markCompactedAway()
private void	open()	Opens reader on this store file.
String	toString()
String	toStringDetailed()	Returns a length description of this StoreFile, suitable for debug output

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Details

LOG

private static final org.slf4j.Logger LOG

MAX_SEQ_ID_KEY

public static final byte[] MAX_SEQ_ID_KEY

Max Sequence ID in FileInfo

MAJOR_COMPACTION_KEY

public static final byte[] MAJOR_COMPACTION_KEY

Major compaction flag in FileInfo

EXCLUDE_FROM_MINOR_COMPACTION_KEY

public static final byte[] EXCLUDE_FROM_MINOR_COMPACTION_KEY

Minor compaction flag in FileInfo

COMPACTION_EVENT_KEY

public static final byte[] COMPACTION_EVENT_KEY

Key for compaction event which contains the compacted storefiles in FileInfo

BLOOM_FILTER_TYPE_KEY

public static final byte[] BLOOM_FILTER_TYPE_KEY

Bloom filter Type in FileInfo

BLOOM_FILTER_PARAM_KEY

public static final byte[] BLOOM_FILTER_PARAM_KEY

Bloom filter param in FileInfo

DELETE_FAMILY_COUNT

public static final byte[] DELETE_FAMILY_COUNT

Delete Family Count in FileInfo

LAST_BLOOM_KEY

public static final byte[] LAST_BLOOM_KEY

Last Bloom filter key in FileInfo

TIMERANGE_KEY

public static final byte[] TIMERANGE_KEY

Key for Timerange information in metadata

EARLIEST_PUT_TS

public static final byte[] EARLIEST_PUT_TS

Key for timestamp of earliest-put in metadata

MOB_CELLS_COUNT

public static final byte[] MOB_CELLS_COUNT

Key for the number of mob cells in metadata

NULL_VALUE

public static final byte[] NULL_VALUE

Null data

MOB_FILE_REFS

public static final byte[] MOB_FILE_REFS

Key for the list of MOB file references

BULKLOAD_TASK_KEY

public static final byte[] BULKLOAD_TASK_KEY

Meta key set when store file is a result of a bulk load

BULKLOAD_TIME_KEY

public static final byte[] BULKLOAD_TIME_KEY

SKIP_RESET_SEQ_ID

public static final byte[] SKIP_RESET_SEQ_ID

Key for skipping resetting sequence id in metadata. For bulk loaded hfiles, the scanner resets the cell seqId with the latest one, if this metadata is set as true, the reset is skipped.

fileInfo

private final StoreFileInfo fileInfo

initialReader

private volatile StoreFileReader initialReader

initialReaderBlockDistribution

private volatile InputStreamBlockDistribution initialReaderBlockDistribution

cacheConf

private final CacheConfig cacheConf

metrics

private final BloomFilterMetrics metrics

compactedAway

private volatile boolean compactedAway

sequenceid

private long sequenceid

maxMemstoreTS

private long maxMemstoreTS

firstKey

private Optional<Cell> firstKey

lastKey

private Optional<Cell> lastKey

comparator

private CellComparator comparator

majorCompaction

private AtomicBoolean majorCompaction

excludeFromMinorCompaction

private boolean excludeFromMinorCompaction

compactedStoreFiles

private final Set<String> compactedStoreFiles

metadataMap

private Map<byte[],byte[]> metadataMap

Map of the metadata entries in the corresponding HFile. Populated when Reader is opened after which it is not modified again.

cfBloomType

private final BloomType cfBloomType

Bloom filter type specified in column family configuration. Does not necessarily correspond to the Bloom filter type present in the HFile.

Constructor Details

HStoreFile

public HStoreFile(org.apache.hadoop.fs.FileSystem fs,
 org.apache.hadoop.fs.Path p,
 org.apache.hadoop.conf.Configuration conf,
 CacheConfig cacheConf,
 BloomType cfBloomType,
 boolean primaryReplica)
           throws IOException

Constructor, loads a reader and it's indices, etc. May allocate a substantial amount of ram depending on the underlying files (10-20MB?).

Parameters:

fs - The current file system to use.

p - The path of the file.

conf - The current configuration.

cacheConf - The cache configuration and block cache reference.

cfBloomType - The bloom type to use for this store file as specified by column family configuration. This may or may not be the same as the Bloom filter type actually present in the HFile, because column family configuration might change. If this is BloomType.NONE, the existing Bloom filter is ignored.

primaryReplica - true if this is a store file for primary replica, otherwise false.

Throws:

IOException

HStoreFile

public HStoreFile(StoreFileInfo fileInfo,
 BloomType cfBloomType,
 CacheConfig cacheConf)

Constructor, loads a reader and it's indices, etc. May allocate a substantial amount of ram depending on the underlying files (10-20MB?).

Parameters:

fileInfo - The store file information.

cfBloomType - The bloom type to use for this store file as specified by column family configuration. This may or may not be the same as the Bloom filter type actually present in the HFile, because column family configuration might change. If this is BloomType.NONE, the existing Bloom filter is ignored.

cacheConf - The cache configuration and block cache reference.

HStoreFile

public HStoreFile(StoreFileInfo fileInfo,
 BloomType cfBloomType,
 CacheConfig cacheConf,
 BloomFilterMetrics metrics)

Constructor, loads a reader and it's indices, etc. May allocate a substantial amount of ram depending on the underlying files (10-20MB?).

Parameters:

fileInfo - The store file information.

cfBloomType - The bloom type to use for this store file as specified by column family configuration. This may or may not be the same as the Bloom filter type actually present in the HFile, because column family configuration might change. If this is BloomType.NONE, the existing Bloom filter is ignored.

cacheConf - The cache configuration and block cache reference.

metrics - Tracks bloom filter requests and results. May be null.

Method Details

getCacheConf

public CacheConfig getCacheConf()

getFirstKey

public Optional<Cell> getFirstKey()

Description copied from interface: StoreFile

Get the first key in this store file.

Specified by:

getFirstKey in interface StoreFile

getLastKey

public Optional<Cell> getLastKey()

Description copied from interface: StoreFile

Get the last key in this store file.

Specified by:

getLastKey in interface StoreFile

getComparator

public CellComparator getComparator()

Description copied from interface: StoreFile

Get the comparator for comparing two cells.

Specified by:

getComparator in interface StoreFile

getMaxMemStoreTS

public long getMaxMemStoreTS()

Description copied from interface: StoreFile

Get max of the MemstoreTS in the KV's in this store file.

Specified by:

getMaxMemStoreTS in interface StoreFile

getFileInfo

public StoreFileInfo getFileInfo()

Returns:

the StoreFile object associated to this StoreFile. null if the StoreFile is not a reference.

getPath

public org.apache.hadoop.fs.Path getPath()

Description copied from interface: StoreFile

Returns Path or null if this StoreFile was made with a Stream.

Specified by:

getPath in interface StoreFile

getEncodedPath

public org.apache.hadoop.fs.Path getEncodedPath()

Description copied from interface: StoreFile

Returns Encoded Path if this StoreFile was made with a Stream.

Specified by:

getEncodedPath in interface StoreFile

getQualifiedPath

public org.apache.hadoop.fs.Path getQualifiedPath()

Description copied from interface: StoreFile

Returns Returns the qualified path of this StoreFile

Specified by:

getQualifiedPath in interface StoreFile

isReference

public boolean isReference()

Description copied from interface: StoreFile

Returns True if this is a StoreFile Reference.

Specified by:

isReference in interface StoreFile

isHFile

public boolean isHFile()

Description copied from interface: StoreFile

Returns True if this is HFile.

Specified by:

isHFile in interface StoreFile

isMajorCompactionResult

public boolean isMajorCompactionResult()

Description copied from interface: StoreFile

Returns True if this file was made by a major compaction.

Specified by:

isMajorCompactionResult in interface StoreFile

excludeFromMinorCompaction

public boolean excludeFromMinorCompaction()

Description copied from interface: StoreFile

Returns True if this file should not be part of a minor compaction.

Specified by:

excludeFromMinorCompaction in interface StoreFile

getMaxSequenceId

public long getMaxSequenceId()

Description copied from interface: StoreFile

Returns This files maximum edit sequence id.

Specified by:

getMaxSequenceId in interface StoreFile

getModificationTimeStamp

public long getModificationTimeStamp()
                              throws IOException

Description copied from interface: StoreFile

Get the modification time of this store file. Usually will access the file system so throws IOException.

Specified by:

getModificationTimeStamp in interface StoreFile

Throws:

IOException

See Also:

• StoreFile.getModificationTimestamp()

getModificationTimestamp

public long getModificationTimestamp()
                              throws IOException

Description copied from interface: StoreFile

Get the modification time of this store file. Usually will access the file system so throws IOException.

Specified by:

getModificationTimestamp in interface StoreFile

Throws:

IOException

getMetadataValue

public byte[] getMetadataValue(byte[] key)

Parameters:

key - to look up

Returns:

value associated with the metadata key

isBulkLoadResult

public boolean isBulkLoadResult()

Description copied from interface: StoreFile

Check if this storefile was created by bulk load. When a hfile is bulk loaded into HBase, we append '_SeqId_<id-when-loaded>' to the hfile name, unless "hbase.mapreduce.bulkload.assign.sequenceNumbers" is explicitly turned off. If "hbase.mapreduce.bulkload.assign.sequenceNumbers" is turned off, fall back to BULKLOAD_TIME_KEY.

Specified by:

isBulkLoadResult in interface StoreFile

Returns:

true if this storefile was created by bulk load.

isCompactedAway

public boolean isCompactedAway()

getRefCount

public int getRefCount()

isReferencedInReads

public boolean isReferencedInReads()

Returns true if the file is still used in reads

getBulkLoadTimestamp

public OptionalLong getBulkLoadTimestamp()

Description copied from interface: StoreFile

Return the timestamp at which this bulk load file was generated.

Specified by:

getBulkLoadTimestamp in interface StoreFile

getHDFSBlockDistribution

public HDFSBlocksDistribution getHDFSBlockDistribution()

Returns:

the cached value of HDFS blocks distribution. The cached value is calculated when store file is opened.

open

private void open()
           throws IOException

Opens reader on this store file. Called by Constructor.

Throws:

IOException

See Also:

• closeStoreFile(boolean)

initReader

public void initReader()
                throws IOException

Initialize the reader used for pread.

Throws:

IOException

createStreamReader

private StoreFileReader createStreamReader(boolean canUseDropBehind)
                                    throws IOException

Throws:

IOException

getPreadScanner

public StoreFileScanner getPreadScanner(boolean cacheBlocks,
 long readPt,
 long scannerOrder,
 boolean canOptimizeForNonNullColumn)

Get a scanner which uses pread.

Must be called after initReader.

getStreamScanner

public StoreFileScanner getStreamScanner(boolean canUseDropBehind,
 boolean cacheBlocks,
 boolean isCompaction,
 long readPt,
 long scannerOrder,
 boolean canOptimizeForNonNullColumn)
                                  throws IOException

Get a scanner which uses streaming read.

Must be called after initReader.

Throws:

IOException

getReader

public StoreFileReader getReader()

Returns:

Current reader. Must call initReader first else returns null.

See Also:

• initReader()

closeStoreFile

public void closeStoreFile(boolean evictOnClose)
                    throws IOException

Parameters:

evictOnClose - whether to evict blocks belonging to this file

Throws:

IOException

deleteStoreFile

public void deleteStoreFile()
                     throws IOException

Delete this file

Throws:

IOException

markCompactedAway

public void markCompactedAway()

toString

public String toString()

Overrides:

toString in class Object

toStringDetailed

public String toStringDetailed()

Description copied from interface: StoreFile

Returns a length description of this StoreFile, suitable for debug output

Specified by:

toStringDetailed in interface StoreFile

isSkipResetSeqId

private boolean isSkipResetSeqId(byte[] skipResetSeqId)

Gets whether to skip resetting the sequence id for cells.

Parameters:

skipResetSeqId - The byte array of boolean.

Returns:

Whether to skip resetting the sequence id.

getMinimumTimestamp

public OptionalLong getMinimumTimestamp()

Description copied from interface: StoreFile

Get the min timestamp of all the cells in the store file.

Specified by:

getMinimumTimestamp in interface StoreFile

getMaximumTimestamp

public OptionalLong getMaximumTimestamp()

Description copied from interface: StoreFile

Get the max timestamp of all the cells in the store file.

Specified by:

getMaximumTimestamp in interface StoreFile

getCompactedStoreFiles

Set<String> getCompactedStoreFiles()

increaseRefCount

long increaseRefCount()

decreaseRefCount

long decreaseRefCount()

increaseStoreFilesRefeCount

static void increaseStoreFilesRefeCount(Collection<HStoreFile> storeFiles)

decreaseStoreFilesRefeCount

static void decreaseStoreFilesRefeCount(Collection<HStoreFile> storeFiles)