Package org.apache.hadoop.hbase.regionserver.compactions

Class CustomDateTieredCompactionPolicy

java.lang.Object

org.apache.hadoop.hbase.regionserver.compactions.CompactionPolicy

org.apache.hadoop.hbase.regionserver.compactions.SortedCompactionPolicy

org.apache.hadoop.hbase.regionserver.compactions.DateTieredCompactionPolicy

org.apache.hadoop.hbase.regionserver.compactions.CustomDateTieredCompactionPolicy

@Private
public class CustomDateTieredCompactionPolicy
extends DateTieredCompactionPolicy

Custom implementation of DateTieredCompactionPolicy that calculates compaction boundaries based on the hbase.hstore.compaction.date.tiered.custom.age.limit.millis configuration property and the TIERING_CELL_MIN/TIERING_CELL_MAX stored on metadata of each store file. This policy would produce either one or two tiers: - One tier if either all files data age are older than the configured age limit or all files data age are younger than the configured age limit. - Two tiers if files have both younger and older data than the configured age limit.

Field Summary

Fields

Modifier and Type	Field	Description
static final String	AGE_LIMIT_MILLIS
private long	cutOffTimestamp
static final long	DEFAULT_AGE_LIMIT_MILLIS
private static final org.slf4j.Logger	LOG

Fields inherited from class org.apache.hadoop.hbase.regionserver.compactions.DateTieredCompactionPolicy

compactionPolicyPerWindow

Fields inherited from class org.apache.hadoop.hbase.regionserver.compactions.CompactionPolicy

comConf, storeConfigInfo

Constructor Summary

Constructors

Constructor	Description
CustomDateTieredCompactionPolicy(org.apache.hadoop.conf.Configuration conf, StoreConfigInformation storeConfigInfo)

Method Summary

Modifier and Type	Method	Description
protected List<Long>	getCompactBoundariesForMajor(Collection<HStoreFile> filesToCompact, long now)	Return a list of boundaries for multiple compaction output in ascending order.
CompactionRequestImpl	selectMinorCompaction(ArrayList<HStoreFile> candidateSelection, boolean mayUseOffPeak, boolean mayBeStuck)	We receive store files sorted in ascending order by seqId then scan the list of files.
boolean	shouldPerformMajorCompaction(Collection<HStoreFile> filesToCompact)

Methods inherited from class org.apache.hadoop.hbase.regionserver.compactions.DateTieredCompactionPolicy

checkBlockLocality, checkForTtl, createCompactionRequest, isMajorCompactionTime, isMajorOrBulkloadResult, needsCompaction, selectMajorCompaction

Methods inherited from class org.apache.hadoop.hbase.regionserver.compactions.SortedCompactionPolicy

checkMinFilesCriteria, filterBulk, getCurrentEligibleFiles, getNextMajorCompactTime, preSelectCompactionForCoprocessor, removeExcessFiles, selectCompaction, skipLargeFiles, throttleCompaction

Methods inherited from class org.apache.hadoop.hbase.regionserver.compactions.CompactionPolicy

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

AGE_LIMIT_MILLIS

public static final String AGE_LIMIT_MILLIS

See Also:

• Constant Field Values

DEFAULT_AGE_LIMIT_MILLIS

public static final long DEFAULT_AGE_LIMIT_MILLIS

See Also:

• Constant Field Values

LOG

private static final org.slf4j.Logger LOG

cutOffTimestamp

private long cutOffTimestamp

Constructor Details

CustomDateTieredCompactionPolicy

public CustomDateTieredCompactionPolicy(org.apache.hadoop.conf.Configuration conf,
 StoreConfigInformation storeConfigInfo)
                                 throws IOException

Throws:

IOException

Method Details

getCompactBoundariesForMajor

protected List<Long> getCompactBoundariesForMajor(Collection<HStoreFile> filesToCompact,
 long now)

Description copied from class: DateTieredCompactionPolicy

Return a list of boundaries for multiple compaction output in ascending order.

Overrides:

getCompactBoundariesForMajor in class DateTieredCompactionPolicy

selectMinorCompaction

public CompactionRequestImpl selectMinorCompaction(ArrayList<HStoreFile> candidateSelection,
 boolean mayUseOffPeak,
 boolean mayBeStuck)
                                            throws IOException

Description copied from class: DateTieredCompactionPolicy

We receive store files sorted in ascending order by seqId then scan the list of files. If the current file has a maxTimestamp older than last known maximum, treat this file as it carries the last known maximum. This way both seqId and timestamp are in the same order. If files carry the same maxTimestamps, they are ordered by seqId. We then reverse the list so they are ordered by seqId and maxTimestamp in descending order and build the time windows. All the out-of-order data into the same compaction windows, guaranteeing contiguous compaction based on sequence id.

Overrides:

selectMinorCompaction in class DateTieredCompactionPolicy

Throws:

IOException

shouldPerformMajorCompaction

public boolean shouldPerformMajorCompaction(Collection<HStoreFile> filesToCompact)
                                     throws IOException

Overrides:

shouldPerformMajorCompaction in class DateTieredCompactionPolicy

Parameters:

filesToCompact - Files to compact. Can be null.

Returns:

True if we should run a major compaction.

Throws:

IOException