Package org.apache.hadoop.hbase.mob

Class DefaultMobStoreCompactor

java.lang.Object

org.apache.hadoop.hbase.regionserver.compactions.Compactor<StoreFileWriter>

org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor

org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor

@Private
public class DefaultMobStoreCompactor
extends DefaultCompactor

Compact passed set of files in the mob-enabled column family.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.hbase.regionserver.compactions.Compactor

Compactor.CellSinkFactory<S>, Compactor.FileDetails, Compactor.InternalScannerFactory

Field Summary

Fields

Modifier and Type	Field	Description
protected final boolean	cacheMobBlocksOnCompaction
(package private) static ThreadLocal<Boolean>	disableIO
protected boolean	ioOptimizedMode
private static final org.slf4j.Logger	LOG
(package private) static ThreadLocal<HashMap<String,Long>>	mobLengthMap
(package private) static ThreadLocal<org.apache.hbase.thirdparty.com.google.common.collect.SetMultimap<TableName,String>>	mobRefSet
protected long	mobSizeThreshold
protected HMobStore	mobStore
private final Compactor.InternalScannerFactory	scannerFactory
(package private) static ThreadLocal<Boolean>	userRequest
private final Compactor.CellSinkFactory<StoreFileWriter>	writerFactory

Fields inherited from class org.apache.hadoop.hbase.regionserver.compactions.Compactor

COMPACTION_PROGRESS_LOG_INTERVAL, compactionKVMax, compactScannerSizeLimit, conf, defaultScannerFactory, dropCacheMajor, dropCacheMinor, keepSeqIdPeriod, MAJOR_COMPACTION_DROP_CACHE, majorCompactionCompression, MINOR_COMPACTION_DROP_CACHE, minorCompactionCompression, store

Constructor Summary

Constructors

Constructor	Description
DefaultMobStoreCompactor(org.apache.hadoop.conf.Configuration conf, HStore store)

Method Summary

Modifier and Type	Method	Description
private void	calculateMobLengthMap(org.apache.hbase.thirdparty.com.google.common.collect.SetMultimap<TableName,String> mobRefs)
private void	clearThreadLocals()
private void	commitOrAbortMobWriter(StoreFileWriter mobFileWriter, long maxSeqId, long mobCells, boolean major)
protected List<org.apache.hadoop.fs.Path>	commitWriter(StoreFileWriter writer, Compactor.FileDetails fd, CompactionRequestImpl request)
List<org.apache.hadoop.fs.Path>	compact(CompactionRequestImpl request, ThroughputController throughputController, User user)	Do a minor/major compaction on an explicit set of storefiles from a Store.
private void	deleteCommittedMobFiles(List<String> fileNames)
protected String	getStoreInfo()
private StoreFileWriter	newMobWriter(Compactor.FileDetails fd, boolean major, Consumer<org.apache.hadoop.fs.Path> writerCreationTracker)
protected boolean	performCompaction(Compactor.FileDetails fd, InternalScanner scanner, CellSink writer, long smallestReadPoint, boolean cleanSeqId, ThroughputController throughputController, CompactionRequestImpl request, CompactionProgress progress)	Performs compaction on a column family with the mob flag enabled.
protected ExtendedCell	resolveMobCell(ExtendedCell reference)	Resolves a MOB reference cell to its backing MOB value and returns an independent, heap-resident copy of the resolved cell.
private StoreFileWriter	switchToNewMobWriter(StoreFileWriter mobFileWriter, Compactor.FileDetails fd, long mobCells, boolean major, CompactionRequestImpl request, List<String> committedMobWriterFileNames)

Methods inherited from class org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor

abortWriter

Methods inherited from class org.apache.hadoop.hbase.regionserver.compactions.Compactor

compact, createParams, createScanner, createScanner, createWriter, createWriter, decorateCells, getProgress, isCompacting

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

LOG

private static final org.slf4j.Logger LOG

mobSizeThreshold

protected long mobSizeThreshold

mobStore

protected HMobStore mobStore

ioOptimizedMode

protected boolean ioOptimizedMode

cacheMobBlocksOnCompaction

protected final boolean cacheMobBlocksOnCompaction

mobRefSet

static ThreadLocal<org.apache.hbase.thirdparty.com.google.common.collect.SetMultimap<TableName,String>> mobRefSet

userRequest

static ThreadLocal<Boolean> userRequest

disableIO

static ThreadLocal<Boolean> disableIO

mobLengthMap

static ThreadLocal<HashMap<String,Long>> mobLengthMap

scannerFactory

private final Compactor.InternalScannerFactory scannerFactory

writerFactory

private final Compactor.CellSinkFactory<StoreFileWriter> writerFactory

Constructor Details

DefaultMobStoreCompactor

public DefaultMobStoreCompactor(org.apache.hadoop.conf.Configuration conf,
 HStore store)

Method Details

resolveMobCell

protected ExtendedCell resolveMobCell(ExtendedCell reference)
                               throws IOException

Resolves a MOB reference cell to its backing MOB value and returns an independent, heap-resident copy of the resolved cell.

A MOB cell resolved from a MOB file is backed by a StoreFileScanner; closing the MobCell closes that scanner and may release/recycle the NIO buffers referenced by the returned cell. We close the MobCell here to avoid leaking scanners/buffers while compacting many reference cells.

The KeyValueUtil.copyToNewKeyValue(ExtendedCell) call is required by this ownership model: HFile writers and encoders may retain references to appended cells (e.g. lastCell, firstCellInBlock, and the data block encoder's prevCell) until beforeShipped(). Returning the scanner-backed cell directly would let those later reads access released buffers. Removing this copy would require changing the caller to retain each MobCell and close it only after the writers have shipped their retained references.

Throws:

IOException

compact

public List<org.apache.hadoop.fs.Path> compact(CompactionRequestImpl request,
 ThroughputController throughputController,
 User user)
                                        throws IOException

Description copied from class: DefaultCompactor

Do a minor/major compaction on an explicit set of storefiles from a Store.

Overrides:

compact in class DefaultCompactor

Throws:

IOException

calculateMobLengthMap

private void calculateMobLengthMap(org.apache.hbase.thirdparty.com.google.common.collect.SetMultimap<TableName,String> mobRefs)
                            throws IOException

Parameters:

mobRefs - multimap of original table name -> mob hfile

Throws:

IOException

performCompaction

protected boolean performCompaction(Compactor.FileDetails fd,
 InternalScanner scanner,
 CellSink writer,
 long smallestReadPoint,
 boolean cleanSeqId,
 ThroughputController throughputController,
 CompactionRequestImpl request,
 CompactionProgress progress)
                             throws IOException

Performs compaction on a column family with the mob flag enabled. This works only when MOB compaction is explicitly requested (by User), or by Master There are two modes of a MOB compaction:

• 1. Full mode - when all MOB data for a region is compacted into a single MOB file.
• 2. I/O optimized mode - for use cases with no or infrequent updates/deletes of a
  MOB data. The main idea behind i/o optimized compaction is to limit maximum size of a MOB file produced during compaction and to limit I/O write/read amplification.

The basic algorithm of compaction is the following:
1. If the Put cell has a mob reference tag, the cell's value is the path of the mob file.

1. If the value size of a cell is larger than the threshold, this cell is regarded as a mob, directly copy the (with mob tag) cell into the new store file.
2. Otherwise, retrieve the mob cell from the mob file, and writes a copy of the cell into the new store file.

2. If the Put cell doesn't have a reference tag.

1. If the value size of a cell is larger than the threshold, this cell is regarded as a mob, write this cell to a mob file, and write the path of this mob file to the store file.
2. Otherwise, directly write this cell into the store file.

Overrides:

performCompaction in class Compactor<StoreFileWriter>

Parameters:

fd - File details

scanner - Where to read from.

writer - Where to write to.

smallestReadPoint - Smallest read point.

cleanSeqId - When true, remove seqId(used to be mvcc) value which is <= smallestReadPoint

throughputController - The compaction throughput controller.

request - compaction request.

progress - Progress reporter.

Returns:

Whether compaction ended; false if it was interrupted for any reason.

Throws:

IOException

getStoreInfo

protected String getStoreInfo()

clearThreadLocals

private void clearThreadLocals()

newMobWriter

private StoreFileWriter newMobWriter(Compactor.FileDetails fd,
 boolean major,
 Consumer<org.apache.hadoop.fs.Path> writerCreationTracker)
                              throws IOException

Throws:

IOException

commitOrAbortMobWriter

private void commitOrAbortMobWriter(StoreFileWriter mobFileWriter,
 long maxSeqId,
 long mobCells,
 boolean major)
                             throws IOException

Throws:

IOException

commitWriter

protected List<org.apache.hadoop.fs.Path> commitWriter(StoreFileWriter writer,
 Compactor.FileDetails fd,
 CompactionRequestImpl request)
                                                throws IOException

Overrides:

commitWriter in class DefaultCompactor

Throws:

IOException

switchToNewMobWriter

private StoreFileWriter switchToNewMobWriter(StoreFileWriter mobFileWriter,
 Compactor.FileDetails fd,
 long mobCells,
 boolean major,
 CompactionRequestImpl request,
 List<String> committedMobWriterFileNames)
                                      throws IOException

Throws:

IOException

deleteCommittedMobFiles

private void deleteCommittedMobFiles(List<String> fileNames)