org.apache.hadoop.hbase.regionserver

Class StoreFileWriter

java.lang.Object

org.apache.hadoop.hbase.regionserver.StoreFileWriter

All Implemented Interfaces:

CellSink, ShipperListener

@InterfaceAudience.Private
public class StoreFileWriter
extends Object
implements CellSink, ShipperListener

A StoreFile writer. Use this to read/write HBase Store Files. It is package local because it is an implementation detail of the HBase regionserver.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	StoreFileWriter.Builder

Field Summary

Fields

Modifier and Type	Field and Description
private BloomContext	bloomContext
private byte[]	bloomParam
private BloomType	bloomType
private Supplier<Collection<HStoreFile>>	compactedFilesSupplier
private static Pattern	dash
private BloomContext	deleteFamilyBloomContext
private BloomFilterWriter	deleteFamilyBloomFilterWriter
private long	deleteFamilyCnt
private long	earliestPutTs
private BloomFilterWriter	generalBloomFilterWriter
private static org.slf4j.Logger	LOG
private TimeRangeTracker	timeRangeTracker
protected HFile.Writer	writer

Constructor Summary

Constructors

Modifier	Constructor and Description
private	StoreFileWriter(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration conf, CacheConfig cacheConf, BloomType bloomType, long maxKeys, InetSocketAddress[] favoredNodes, HFileContext fileContext, boolean shouldDropCacheBehind, Supplier<Collection<HStoreFile>> compactedFilesSupplier) Creates an HFile.Writer that also write helpful meta data.

Method Summary

Modifier and Type	Method and Description
void	append(Cell cell) Append the given cell
private void	appendDeleteFamilyBloomFilter(Cell cell)
void	appendFileInfo(byte[] key, byte[] value)
private void	appendGeneralBloomfilter(Cell cell)
void	appendMetadata(long maxSequenceId, boolean majorCompaction) Writes meta data.
void	appendMetadata(long maxSequenceId, boolean majorCompaction, Collection<HStoreFile> storeFiles) Writes meta data.
void	appendMetadata(long maxSequenceId, boolean majorCompaction, long mobCellsCount) Writes meta data.
void	appendMobMetadata(org.apache.hbase.thirdparty.com.google.common.collect.SetMultimap<TableName,String> mobRefSet) Appends MOB - specific metadata (even if it is empty)
void	appendTrackedTimestampsToMetadata() Add TimestampRange and earliest put timestamp to Metadata
void	beforeShipped() The action that needs to be performed before Shipper.shipped() is performed n
void	close()
private boolean	closeBloomFilter(BloomFilterWriter bfw)
private boolean	closeDeleteFamilyBloomFilter()
private boolean	closeGeneralBloomFilter()
(package private) BloomFilterWriter	getGeneralBloomWriter() For unit testing only.
(package private) HFile.Writer	getHFileWriter() For use in testing.
org.apache.hadoop.fs.Path	getPath()
long	getPos()
(package private) static org.apache.hadoop.fs.Path	getUniqueFile(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path dir)
boolean	hasGeneralBloom()
private byte[]	toCompactionEventTrackerBytes(Collection<HStoreFile> storeFiles) Used when write HStoreFile.COMPACTION_EVENT_KEY to new file's file info.
void	trackTimestamps(Cell cell) Record the earlest Put timestamp.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

private static final org.slf4j.Logger LOG

dash

private static final Pattern dash

generalBloomFilterWriter

private final BloomFilterWriter generalBloomFilterWriter

deleteFamilyBloomFilterWriter

private final BloomFilterWriter deleteFamilyBloomFilterWriter

bloomType

private final BloomType bloomType

bloomParam

private byte[] bloomParam

earliestPutTs

private long earliestPutTs

deleteFamilyCnt

private long deleteFamilyCnt

bloomContext

private BloomContext bloomContext

deleteFamilyBloomContext

private BloomContext deleteFamilyBloomContext

timeRangeTracker

private final TimeRangeTracker timeRangeTracker

compactedFilesSupplier

private final Supplier<Collection<HStoreFile>> compactedFilesSupplier

writer

protected HFile.Writer writer

Constructor Detail

StoreFileWriter

private StoreFileWriter(org.apache.hadoop.fs.FileSystem fs,
                        org.apache.hadoop.fs.Path path,
                        org.apache.hadoop.conf.Configuration conf,
                        CacheConfig cacheConf,
                        BloomType bloomType,
                        long maxKeys,
                        InetSocketAddress[] favoredNodes,
                        HFileContext fileContext,
                        boolean shouldDropCacheBehind,
                        Supplier<Collection<HStoreFile>> compactedFilesSupplier)
                 throws IOException

Creates an HFile.Writer that also write helpful meta data.

Parameters:

fs - file system to write to

path - file name to create

conf - user configuration

bloomType - bloom filter setting

maxKeys - the expected maximum number of keys to be added. Was used for Bloom filter size in HFile format version 1.

favoredNodes - an array of favored nodes or possibly null

fileContext - The HFile context

shouldDropCacheBehind - Drop pages written to page cache after writing the store file.

compactedFilesSupplier - Returns the HStore compacted files which not archived

Throws:

IOException - problem writing to FS

Method Detail

getPos

public long getPos()
            throws IOException

Throws:

IOException

appendMetadata

public void appendMetadata(long maxSequenceId,
                           boolean majorCompaction)
                    throws IOException

Writes meta data. Call before close() since its written as meta data to this file.

Parameters:

maxSequenceId - Maximum sequence id.

majorCompaction - True if this file is product of a major compaction

Throws:

IOException - problem writing to FS

appendMetadata

public void appendMetadata(long maxSequenceId,
                           boolean majorCompaction,
                           Collection<HStoreFile> storeFiles)
                    throws IOException

Writes meta data. Call before close() since its written as meta data to this file.

Parameters:

maxSequenceId - Maximum sequence id.

majorCompaction - True if this file is product of a major compaction

storeFiles - The compacted store files to generate this new file

Throws:

IOException - problem writing to FS

toCompactionEventTrackerBytes

private byte[] toCompactionEventTrackerBytes(Collection<HStoreFile> storeFiles)

Used when write HStoreFile.COMPACTION_EVENT_KEY to new file's file info. The compacted store files's name is needed. But if the compacted store file is a result of compaction, it's compacted files which still not archived is needed, too. And don't need to add compacted files recursively. If file A, B, C compacted to new file D, and file D compacted to new file E, will write A, B, C, D to file E's compacted files. So if file E compacted to new file F, will add E to F's compacted files first, then add E's compacted files: A, B, C, D to it. And no need to add D's compacted file, as D's compacted files has been in E's compacted files, too. See HBASE-20724 for more details.

Parameters:

storeFiles - The compacted store files to generate this new file

Returns:

bytes of CompactionEventTracker

appendMetadata

public void appendMetadata(long maxSequenceId,
                           boolean majorCompaction,
                           long mobCellsCount)
                    throws IOException

Writes meta data. Call before close() since its written as meta data to this file.

Parameters:

maxSequenceId - Maximum sequence id.

majorCompaction - True if this file is product of a major compaction

mobCellsCount - The number of mob cells.

Throws:

IOException - problem writing to FS

appendMobMetadata

public void appendMobMetadata(org.apache.hbase.thirdparty.com.google.common.collect.SetMultimap<TableName,String> mobRefSet)
                       throws IOException

Appends MOB - specific metadata (even if it is empty)

Parameters:

mobRefSet - - original table -> set of MOB file names

Throws:

IOException - problem writing to FS

appendTrackedTimestampsToMetadata

public void appendTrackedTimestampsToMetadata()
                                       throws IOException

Add TimestampRange and earliest put timestamp to Metadata

Throws:

IOException

trackTimestamps

public void trackTimestamps(Cell cell)

Record the earlest Put timestamp. If the timeRangeTracker is not set, update TimeRangeTracker to include the timestamp of this key

appendGeneralBloomfilter

private void appendGeneralBloomfilter(Cell cell)
                               throws IOException

Throws:

IOException

appendDeleteFamilyBloomFilter

private void appendDeleteFamilyBloomFilter(Cell cell)
                                    throws IOException

Throws:

IOException

append

public void append(Cell cell)
            throws IOException

Description copied from interface: CellSink

Append the given cell

Specified by:

append in interface CellSink

Parameters:

cell - the cell to be added n

Throws:

IOException

beforeShipped

public void beforeShipped()
                   throws IOException

Description copied from interface: ShipperListener

The action that needs to be performed before Shipper.shipped() is performed n

Specified by:

beforeShipped in interface ShipperListener

Throws:

IOException

getPath

public org.apache.hadoop.fs.Path getPath()

hasGeneralBloom

public boolean hasGeneralBloom()

getGeneralBloomWriter

BloomFilterWriter getGeneralBloomWriter()

For unit testing only.

Returns:

the Bloom filter used by this writer.

closeBloomFilter

private boolean closeBloomFilter(BloomFilterWriter bfw)
                          throws IOException

Throws:

IOException

closeGeneralBloomFilter

private boolean closeGeneralBloomFilter()
                                 throws IOException

Throws:

IOException

closeDeleteFamilyBloomFilter

private boolean closeDeleteFamilyBloomFilter()
                                      throws IOException

Throws:

IOException

close

public void close()
           throws IOException

Throws:

IOException

appendFileInfo

public void appendFileInfo(byte[] key,
                           byte[] value)
                    throws IOException

Throws:

IOException

getHFileWriter

HFile.Writer getHFileWriter()

For use in testing.

getUniqueFile

static org.apache.hadoop.fs.Path getUniqueFile(org.apache.hadoop.fs.FileSystem fs,
                                               org.apache.hadoop.fs.Path dir)
                                        throws IOException

Parameters:

dir - Directory to create file in.

Returns:

random filename inside passed dir

Throws:

IOException