org.apache.hadoop.hbase.regionserver

Class HMobStore

java.lang.Object

org.apache.hadoop.hbase.regionserver.HStore

org.apache.hadoop.hbase.regionserver.HMobStore

All Implemented Interfaces:

ConfigurationObserver, PropagatingConfigurationObserver, HeapSize, Store, StoreConfigInformation

@InterfaceAudience.Private
public class HMobStore
extends HStore

The store implementation to save MOBs (medium objects), it extends the HStore. When a descriptor of a column family has the value "IS_MOB", it means this column family is a mob one. When a HRegion instantiate a store for this column family, the HMobStore is created. HMobStore is almost the same with the HStore except using different types of scanners. In the method of getScanner, the MobStoreScanner and MobReversedStoreScanner are returned. In these scanners, a additional seeks in the mob files should be performed after the seek to HBase is done. The store implements how we save MOBs by extending HStore. When a descriptor of a column family has the value "IS_MOB", it means this column family is a mob one. When a HRegion instantiate a store for this column family, the HMobStore is created. HMobStore is almost the same with the HStore except using different types of scanners. In the method of getScanner, the MobStoreScanner and MobReversedStoreScanner are returned. In these scanners, a additional seeks in the mob files should be performed after the seek in HBase is done.

Field Summary

Fields

Modifier and Type	Field and Description
private AtomicLong	cellsCountCompactedFromMob
private AtomicLong	cellsCountCompactedToMob
private AtomicLong	cellsSizeCompactedFromMob
private AtomicLong	cellsSizeCompactedToMob
private org.apache.hadoop.fs.Path	homePath
private IdLock	keyLock
private static org.slf4j.Logger	LOG
private Map<TableName,List<org.apache.hadoop.fs.Path>>	map
private org.apache.hadoop.fs.Path	mobFamilyPath
private MobFileCache	mobFileCache
private AtomicLong	mobFlushCount
private AtomicLong	mobFlushedCellsCount
private AtomicLong	mobFlushedCellsSize
private AtomicLong	mobScanCellsCount
private AtomicLong	mobScanCellsSize
private byte[]	refCellTags

Fields inherited from class org.apache.hadoop.hbase.regionserver.HStore

archiveLock, BLOCK_STORAGE_POLICY_KEY, BLOCKING_STOREFILES_KEY, COMPACTCHECKER_INTERVAL_MULTIPLIER_KEY, conf, DEEP_OVERHEAD, DEFAULT_BLOCK_STORAGE_POLICY, DEFAULT_BLOCKING_STOREFILE_COUNT, DEFAULT_COMPACTCHECKER_INTERVAL_MULTIPLIER, FIXED_OVERHEAD, forceMajor, memstore, MEMSTORE_CLASS_NAME, storeEngine

Fields inherited from interface org.apache.hadoop.hbase.regionserver.Store

NO_PRIORITY, PRIORITY_USER

Constructor Summary

Constructors

Constructor and Description
HMobStore(HRegion region, ColumnFamilyDescriptor family, org.apache.hadoop.conf.Configuration confParam, boolean warmup)

Method Summary

Modifier and Type	Method and Description
void	commitFile(org.apache.hadoop.fs.Path sourceFile, org.apache.hadoop.fs.Path targetPath) Commits the mob file.
protected KeyValueScanner	createScanner(Scan scan, ScanInfo scanInfo, NavigableSet<byte[]> targetCols, long readPt) Gets the MobStoreScanner or MobReversedStoreScanner.
protected StoreEngine<?,?,?,?>	createStoreEngine(HStore store, org.apache.hadoop.conf.Configuration conf, CellComparator cellComparator) Creates the mob store engine.
StoreFileWriter	createWriter(Date date, long maxKeyCount, Compression.Algorithm compression, byte[] startKey, boolean isCompaction, Consumer<org.apache.hadoop.fs.Path> writerCreationTracker) Creates the writer for the mob file in the mob family directory.
StoreFileWriter	createWriterInTmp(Date date, long maxKeyCount, Compression.Algorithm compression, byte[] startKey, boolean isCompaction) Creates the writer for the mob file in temp directory.
StoreFileWriter	createWriterInTmp(MobFileName mobFileName, org.apache.hadoop.fs.Path basePath, long maxKeyCount, Compression.Algorithm compression, boolean isCompaction, Consumer<org.apache.hadoop.fs.Path> writerCreationTracker) Creates the writer for the mob file in temp directory.
StoreFileWriter	createWriterInTmp(String date, org.apache.hadoop.fs.Path basePath, long maxKeyCount, Compression.Algorithm compression, byte[] startKey, boolean isCompaction, Consumer<org.apache.hadoop.fs.Path> writerCreationTracker) Creates the writer for the mob file in temp directory.
long	getCellsCountCompactedFromMob()
long	getCellsCountCompactedToMob()
long	getCellsSizeCompactedFromMob()
long	getCellsSizeCompactedToMob()
org.apache.hadoop.conf.Configuration	getConfiguration() Gets current config.
List<org.apache.hadoop.fs.Path>	getLocations(TableName tableName)
long	getMobFlushCount()
long	getMobFlushedCellsCount()
long	getMobFlushedCellsSize()
long	getMobScanCellsCount()
long	getMobScanCellsSize()
org.apache.hadoop.fs.Path	getPath() Gets the mob file path.
byte[]	getRefCellTags()
private org.apache.hadoop.fs.Path	getTempDir() Gets the temp directory.
private MobCell	readCell(List<org.apache.hadoop.fs.Path> locations, String fileName, Cell search, boolean cacheMobBlocks, long readPt, boolean readEmptyValueOnMobCellMiss) Reads the cell from a mob file.
MobCell	resolve(Cell reference, boolean cacheBlocks) Reads the cell from the mob file, and the read point does not count.
MobCell	resolve(Cell reference, boolean cacheBlocks, boolean readEmptyValueOnMobCellMiss) Reads the cell from the mob file with readEmptyValueOnMobCellMiss
MobCell	resolve(Cell reference, boolean cacheBlocks, long readPt, boolean readEmptyValueOnMobCellMiss) Reads the cell from the mob file.
void	updateCellsCountCompactedFromMob(long count)
void	updateCellsCountCompactedToMob(long count)
void	updateCellsSizeCompactedFromMob(long size)
void	updateCellsSizeCompactedToMob(long size)
void	updateMobFlushCount()
void	updateMobFlushedCellsCount(long count)
void	updateMobFlushedCellsSize(long size)
void	updateMobScanCellsCount(long count)
void	updateMobScanCellsSize(long size)
private void	validateMobFile(org.apache.hadoop.fs.Path path) Validates a mob file by opening and closing it.

Methods inherited from class org.apache.hadoop.hbase.regionserver.HStore

add, add, addChangedReaderObserver, areWritesEnabled, assertBulkLoadHFileOk, bulkLoadHFile, bulkLoadHFile, cancelRequestedCompaction, canSplit, close, closeAndArchiveCompactedFiles, compact, createCacheConf, createFlushContext, deleteChangedReaderObserver, deregisterChildren, determineTTLFromFamily, doCompaction, finishCompactionRequest, flushCache, getAvgStoreFileAge, getBlockingFileCount, getBloomFilterEligibleRequestsCount, getBloomFilterNegativeResultsCount, getBloomFilterRequestsCount, getCacheConfig, getColumnFamilyDescriptor, getColumnFamilyName, getCompactedCellsCount, getCompactedCellsSize, getCompactedFiles, getCompactedFilesCount, getCompactionCheckMultiplier, getCompactionPressure, getCompactionProgress, getCompactPriority, getComparator, getCoprocessorHost, getCurrentParallelPutCount, getDataBlockEncoder, getFileSystem, getFlushableSize, getFlushedCellsCount, getFlushedCellsSize, getFlushedOutputFileSize, getHFilesSize, getHRegion, getLastCompactSize, getMajorCompactedCellsCount, getMajorCompactedCellsSize, getMaxCompactedStoreFileRefCount, getMaxMemStoreTS, getMaxSequenceId, getMaxStoreFileAge, getMemStoreFlushSize, getMemstoreOnlyRowReadsCount, getMemStoreSize, getMinStoreFileAge, getMixedRowReadsCount, getNumHFiles, getNumReferenceFiles, getOffPeakHours, getReadOnlyConfiguration, getRegionFileSystem, getRegionInfo, getScanInfo, getScanner, getScanners, getScanners, getScanners, getScanners, getSize, getSmallestReadPoint, getSnapshotSize, getSplitPoint, getStoreContext, getStoreEngine, getStorefiles, getStoreFilesBeingWritten, getStorefilesCount, getStoreFileSize, getStorefilesRootLevelIndexSize, getStorefilesSize, getStoreFileTtl, getStoreRefCount, getStoreSizeUncompressed, getTableName, getTotalStaticBloomSize, getTotalStaticIndexSize, hasReferences, hasTooManyStoreFiles, heapSize, isPrimaryReplicaStore, isSloppyMemStore, needsCompaction, onConfigurationChange, postSnapshotOperation, preBulkLoadHFile, preFlushSeqIDEstimation, preSnapshotOperation, recreateScanners, refreshStoreFiles, refreshStoreFiles, refreshStoreSizeAndTotalBytes, registerChildren, replaceStoreFiles, replayCompactionMarker, reportArchivedFilesForQuota, requestCompaction, requestCompaction, setDataBlockEncoderInTest, setScanInfo, shouldPerformMajorCompaction, startReplayingFromWAL, stopReplayingFromWAL, throttleCompaction, timeOfOldestEdit, toString, triggerMajorCompaction, tryCommitRecoveredHFile, updateCompactedMetrics, updateMetricsStore, updateSpaceQuotaAfterFileReplacement, upsert, versionsToReturn

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

LOG

private static final org.slf4j.Logger LOG

mobFileCache

private MobFileCache mobFileCache

homePath

private org.apache.hadoop.fs.Path homePath

mobFamilyPath

private org.apache.hadoop.fs.Path mobFamilyPath

cellsCountCompactedToMob

private AtomicLong cellsCountCompactedToMob

cellsCountCompactedFromMob

private AtomicLong cellsCountCompactedFromMob

cellsSizeCompactedToMob

private AtomicLong cellsSizeCompactedToMob

cellsSizeCompactedFromMob

private AtomicLong cellsSizeCompactedFromMob

mobFlushCount

private AtomicLong mobFlushCount

mobFlushedCellsCount

private AtomicLong mobFlushedCellsCount

mobFlushedCellsSize

private AtomicLong mobFlushedCellsSize

mobScanCellsCount

private AtomicLong mobScanCellsCount

mobScanCellsSize

private AtomicLong mobScanCellsSize

map

private Map<TableName,List<org.apache.hadoop.fs.Path>> map

keyLock

private final IdLock keyLock

refCellTags

private final byte[] refCellTags

Constructor Detail

HMobStore

public HMobStore(HRegion region,
                 ColumnFamilyDescriptor family,
                 org.apache.hadoop.conf.Configuration confParam,
                 boolean warmup)
          throws IOException

Throws:

IOException

Method Detail

getConfiguration

public org.apache.hadoop.conf.Configuration getConfiguration()

Gets current config.

createScanner

protected KeyValueScanner createScanner(Scan scan,
                                        ScanInfo scanInfo,
                                        NavigableSet<byte[]> targetCols,
                                        long readPt)
                                 throws IOException

Gets the MobStoreScanner or MobReversedStoreScanner. In these scanners, a additional seeks in the mob files should be performed after the seek in HBase is done.

Overrides:

createScanner in class HStore

Throws:

IOException

createStoreEngine

protected StoreEngine<?,?,?,?> createStoreEngine(HStore store,
                                                 org.apache.hadoop.conf.Configuration conf,
                                                 CellComparator cellComparator)
                                          throws IOException

Creates the mob store engine.

Overrides:

createStoreEngine in class HStore

Parameters:

store - The store. An unfortunate dependency needed due to it being passed to coprocessors via the compactor.

conf - Store configuration.

cellComparator - KVComparator for storeFileManager.

Returns:

StoreEngine to use.

Throws:

IOException

getTempDir

private org.apache.hadoop.fs.Path getTempDir()

Gets the temp directory.

Returns:

The temp directory.

createWriterInTmp

public StoreFileWriter createWriterInTmp(Date date,
                                         long maxKeyCount,
                                         Compression.Algorithm compression,
                                         byte[] startKey,
                                         boolean isCompaction)
                                  throws IOException

Creates the writer for the mob file in temp directory.

Parameters:

date - The latest date of written cells.

maxKeyCount - The key count.

compression - The compression algorithm.

startKey - The start key.

isCompaction - If the writer is used in compaction.

Returns:

The writer for the mob file. n

Throws:

IOException

createWriter

public StoreFileWriter createWriter(Date date,
                                    long maxKeyCount,
                                    Compression.Algorithm compression,
                                    byte[] startKey,
                                    boolean isCompaction,
                                    Consumer<org.apache.hadoop.fs.Path> writerCreationTracker)
                             throws IOException

Creates the writer for the mob file in the mob family directory.

Parameters:

date - The latest date of written cells.

maxKeyCount - The key count.

compression - The compression algorithm.

startKey - The start key.

isCompaction - If the writer is used in compaction.

Returns:

The writer for the mob file. n

Throws:

IOException

createWriterInTmp

public StoreFileWriter createWriterInTmp(String date,
                                         org.apache.hadoop.fs.Path basePath,
                                         long maxKeyCount,
                                         Compression.Algorithm compression,
                                         byte[] startKey,
                                         boolean isCompaction,
                                         Consumer<org.apache.hadoop.fs.Path> writerCreationTracker)
                                  throws IOException

Creates the writer for the mob file in temp directory.

Parameters:

date - The date string, its format is yyyymmmdd.

basePath - The basic path for a temp directory.

maxKeyCount - The key count.

compression - The compression algorithm.

startKey - The start key.

isCompaction - If the writer is used in compaction.

Returns:

The writer for the mob file. n

Throws:

IOException

createWriterInTmp

public StoreFileWriter createWriterInTmp(MobFileName mobFileName,
                                         org.apache.hadoop.fs.Path basePath,
                                         long maxKeyCount,
                                         Compression.Algorithm compression,
                                         boolean isCompaction,
                                         Consumer<org.apache.hadoop.fs.Path> writerCreationTracker)
                                  throws IOException

Creates the writer for the mob file in temp directory.

Parameters:

mobFileName - The mob file name.

basePath - The basic path for a temp directory.

maxKeyCount - The key count.

compression - The compression algorithm.

isCompaction - If the writer is used in compaction.

Returns:

The writer for the mob file. n

Throws:

IOException

commitFile

public void commitFile(org.apache.hadoop.fs.Path sourceFile,
                       org.apache.hadoop.fs.Path targetPath)
                throws IOException

Commits the mob file.

Parameters:

sourceFile - The source file.

targetPath - The directory path where the source file is renamed to. n

Throws:

IOException

validateMobFile

private void validateMobFile(org.apache.hadoop.fs.Path path)
                      throws IOException

Validates a mob file by opening and closing it.

Parameters:

path - the path to the mob file

Throws:

IOException

resolve

public MobCell resolve(Cell reference,
                       boolean cacheBlocks)
                throws IOException

Reads the cell from the mob file, and the read point does not count. This is used for DefaultMobStoreCompactor where we can read empty value for the missing cell.

Parameters:

reference - The cell found in the HBase, its value is a path to a mob file.

cacheBlocks - Whether the scanner should cache blocks.

Returns:

The cell found in the mob file. n

Throws:

IOException

resolve

public MobCell resolve(Cell reference,
                       boolean cacheBlocks,
                       boolean readEmptyValueOnMobCellMiss)
                throws IOException

Reads the cell from the mob file with readEmptyValueOnMobCellMiss

Parameters:

reference - The cell found in the HBase, its value is a path to a mob file.

cacheBlocks - Whether the scanner should cache blocks.

readEmptyValueOnMobCellMiss - should return empty mob cell if reference can not be resolved.

Returns:

The cell found in the mob file. n

Throws:

IOException

resolve

public MobCell resolve(Cell reference,
                       boolean cacheBlocks,
                       long readPt,
                       boolean readEmptyValueOnMobCellMiss)
                throws IOException

Reads the cell from the mob file.

Parameters:

reference - The cell found in the HBase, its value is a path to a mob file.

cacheBlocks - Whether the scanner should cache blocks.

readPt - the read point.

readEmptyValueOnMobCellMiss - Whether return null value when the mob file is missing or corrupt.

Returns:

The cell found in the mob file. n

Throws:

IOException

getLocations

public List<org.apache.hadoop.fs.Path> getLocations(TableName tableName)
                                             throws IOException

Parameters:

tableName - to look up locations for, can not be null

Returns:

a list of location in order of working dir, archive dir. will not be null.

Throws:

IOException

readCell

private MobCell readCell(List<org.apache.hadoop.fs.Path> locations,
                         String fileName,
                         Cell search,
                         boolean cacheMobBlocks,
                         long readPt,
                         boolean readEmptyValueOnMobCellMiss)
                  throws IOException

Reads the cell from a mob file. The mob file might be located in different directories. 1. The working directory. 2. The archive directory. Reads the cell from the files located in both of the above directories.

Parameters:

locations - The possible locations where the mob files are saved.

fileName - The file to be read.

search - The cell to be searched.

cacheMobBlocks - Whether the scanner should cache blocks.

readPt - the read point.

readEmptyValueOnMobCellMiss - Whether return null value when the mob file is missing or corrupt.

Returns:

The found cell. Null if there's no such a cell. n

Throws:

IOException

getPath

public org.apache.hadoop.fs.Path getPath()

Gets the mob file path.

Returns:

The mob file path.

updateCellsCountCompactedToMob

public void updateCellsCountCompactedToMob(long count)

getCellsCountCompactedToMob

public long getCellsCountCompactedToMob()

updateCellsCountCompactedFromMob

public void updateCellsCountCompactedFromMob(long count)

getCellsCountCompactedFromMob

public long getCellsCountCompactedFromMob()

updateCellsSizeCompactedToMob

public void updateCellsSizeCompactedToMob(long size)

getCellsSizeCompactedToMob

public long getCellsSizeCompactedToMob()

updateCellsSizeCompactedFromMob

public void updateCellsSizeCompactedFromMob(long size)

getCellsSizeCompactedFromMob

public long getCellsSizeCompactedFromMob()

updateMobFlushCount

public void updateMobFlushCount()

getMobFlushCount

public long getMobFlushCount()

updateMobFlushedCellsCount

public void updateMobFlushedCellsCount(long count)

getMobFlushedCellsCount

public long getMobFlushedCellsCount()

updateMobFlushedCellsSize

public void updateMobFlushedCellsSize(long size)

getMobFlushedCellsSize

public long getMobFlushedCellsSize()

updateMobScanCellsCount

public void updateMobScanCellsCount(long count)

getMobScanCellsCount

public long getMobScanCellsCount()

updateMobScanCellsSize

public void updateMobScanCellsSize(long size)

getMobScanCellsSize

public long getMobScanCellsSize()

getRefCellTags

public byte[] getRefCellTags()