org.apache.hadoop.hbase.regionserver.compactions

Class FIFOCompactionPolicy

java.lang.Object

org.apache.hadoop.hbase.regionserver.compactions.CompactionPolicy

org.apache.hadoop.hbase.regionserver.compactions.SortedCompactionPolicy

org.apache.hadoop.hbase.regionserver.compactions.RatioBasedCompactionPolicy

org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy

org.apache.hadoop.hbase.regionserver.compactions.FIFOCompactionPolicy

@InterfaceAudience.Private
public class FIFOCompactionPolicy
extends ExploringCompactionPolicy

FIFO compaction policy selects only files which have all cells expired. The column family MUST have non-default TTL. One of the use cases for this policy is when we need to store raw data which will be post-processed later and discarded completely after quite short period of time. Raw time-series vs. time-based roll up aggregates and compacted time-series. We collect raw time-series and store them into CF with FIFO compaction policy, periodically we run task which creates roll up aggregates and compacts time-series, the original raw data can be discarded after that.

Field Summary

Fields

Modifier and Type	Field and Description
private static org.slf4j.Logger	LOG

Fields inherited from class org.apache.hadoop.hbase.regionserver.compactions.CompactionPolicy

comConf, storeConfigInfo

Constructor Summary

Constructors

Constructor and Description
FIFOCompactionPolicy(org.apache.hadoop.conf.Configuration conf, StoreConfigInformation storeConfigInfo)

Method Summary

Modifier and Type	Method and Description
private Collection<HStoreFile>	getExpiredStores(Collection<HStoreFile> files, Collection<HStoreFile> filesCompacting)
private boolean	hasExpiredStores(Collection<HStoreFile> files)
private boolean	isEmptyStoreFile(HStoreFile sf) The FIFOCompactionPolicy only choose those TTL expired HFiles as the compaction candidates.
boolean	needsCompaction(Collection<HStoreFile> storeFiles, List<HStoreFile> filesCompacting) A heuristic method to decide whether to schedule a compaction request
CompactionRequestImpl	selectCompaction(Collection<HStoreFile> candidateFiles, List<HStoreFile> filesCompacting, boolean isUserCompaction, boolean mayUseOffPeak, boolean forceMajor)
boolean	shouldPerformMajorCompaction(Collection<HStoreFile> filesToCompact)

Methods inherited from class org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy

applyCompactionPolicy, applyCompactionPolicy

Methods inherited from class org.apache.hadoop.hbase.regionserver.compactions.RatioBasedCompactionPolicy

createCompactionRequest, setMinThreshold

Methods inherited from class org.apache.hadoop.hbase.regionserver.compactions.SortedCompactionPolicy

checkMinFilesCriteria, filterBulk, getCurrentEligibleFiles, getNextMajorCompactTime, preSelectCompactionForCoprocessor, removeExcessFiles, skipLargeFiles, throttleCompaction

Methods inherited from class org.apache.hadoop.hbase.regionserver.compactions.CompactionPolicy

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

private static final org.slf4j.Logger LOG

Constructor Detail

FIFOCompactionPolicy

public FIFOCompactionPolicy(org.apache.hadoop.conf.Configuration conf,
                            StoreConfigInformation storeConfigInfo)

Method Detail

selectCompaction

public CompactionRequestImpl selectCompaction(Collection<HStoreFile> candidateFiles,
                                              List<HStoreFile> filesCompacting,
                                              boolean isUserCompaction,
                                              boolean mayUseOffPeak,
                                              boolean forceMajor)
                                       throws IOException

Overrides:

selectCompaction in class SortedCompactionPolicy

Parameters:

candidateFiles - candidate files, ordered from oldest to newest by seqId. We rely on DefaultStoreFileManager to sort the files by seqId to guarantee contiguous compaction based on seqId for data consistency.

Returns:

subset copy of candidate list that meets compaction criteria

Throws:

IOException

shouldPerformMajorCompaction

public boolean shouldPerformMajorCompaction(Collection<HStoreFile> filesToCompact)
                                     throws IOException

Overrides:

shouldPerformMajorCompaction in class RatioBasedCompactionPolicy

Parameters:

filesToCompact - Files to compact. Can be null.

Returns:

True if we should run a major compaction.

Throws:

IOException

needsCompaction

public boolean needsCompaction(Collection<HStoreFile> storeFiles,
                               List<HStoreFile> filesCompacting)

Description copied from class: RatioBasedCompactionPolicy

A heuristic method to decide whether to schedule a compaction request

Overrides:

needsCompaction in class RatioBasedCompactionPolicy

Parameters:

storeFiles - files in the store.

filesCompacting - files being scheduled to compact.

Returns:

true to schedule a request.

isEmptyStoreFile

private boolean isEmptyStoreFile(HStoreFile sf)

The FIFOCompactionPolicy only choose those TTL expired HFiles as the compaction candidates. So if all HFiles are TTL expired, then the compaction will generate a new empty HFile. While its max timestamp will be Long.MAX_VALUE. If not considered separately, the HFile will never be archived because its TTL will be never expired. So we'll check the empty store file separately. (See HBASE-21504)

hasExpiredStores

private boolean hasExpiredStores(Collection<HStoreFile> files)

getExpiredStores

private Collection<HStoreFile> getExpiredStores(Collection<HStoreFile> files,
                                                Collection<HStoreFile> filesCompacting)