org.apache.hadoop.hbase.regionserver.compactions

Class DateTieredCompactionPolicy

java.lang.Object

org.apache.hadoop.hbase.regionserver.compactions.CompactionPolicy

org.apache.hadoop.hbase.regionserver.compactions.SortedCompactionPolicy

org.apache.hadoop.hbase.regionserver.compactions.DateTieredCompactionPolicy

@InterfaceAudience.LimitedPrivate(value="Configuration")
public class DateTieredCompactionPolicy
extends SortedCompactionPolicy

HBASE-15181 This is a simple implementation of date-based tiered compaction similar to Cassandra's for the following benefits:

1. Improve date-range-based scan by structuring store files in date-based tiered layout.
2. Reduce compaction overhead.
3. Improve TTL efficiency.

Perfect fit for the use cases that:

1. has mostly date-based data write and scan and a focus on the most recent data.

Out-of-order writes are handled gracefully. Time range overlapping among store files is tolerated and the performance impact is minimized. Configuration can be set at hbase-site or overridden at per-table or per-column-family level by hbase shell. Design spec is at https://docs.google.com/document/d/1_AmlNb2N8Us1xICsTeGDLKIqL6T-oHoRLZ323MG_uy8/

Field Summary

Fields

Modifier and Type	Field and Description
private RatioBasedCompactionPolicy	compactionPolicyPerWindow
private static org.slf4j.Logger	LOG
private CompactionWindowFactory	windowFactory

Fields inherited from class org.apache.hadoop.hbase.regionserver.compactions.CompactionPolicy

comConf, storeConfigInfo

Constructor Summary

Constructors

Constructor and Description
DateTieredCompactionPolicy(org.apache.hadoop.conf.Configuration conf, StoreConfigInformation storeConfigInfo)

Method Summary

Modifier and Type	Method and Description
protected CompactionRequestImpl	createCompactionRequest(ArrayList<HStoreFile> candidateSelection, boolean tryingMajor, boolean mayUseOffPeak, boolean mayBeStuck)
private DateTieredCompactionRequest	generateCompactionRequest(ArrayList<HStoreFile> storeFiles, CompactionWindow window, boolean mayUseOffPeak, boolean mayBeStuck, int minThreshold, long now)
private Map<Long,String>	getBoundariesStoragePolicyForMajor(List<Long> boundaries, long now)
private Map<Long,String>	getBoundariesStoragePolicyForMinor(boolean singleOutput, CompactionWindow window, long now)
private List<Long>	getCompactBoundariesForMajor(Collection<HStoreFile> filesToCompact, long now) Return a list of boundaries for multiple compaction output in ascending order.
private static List<Long>	getCompactionBoundariesForMinor(CompactionWindow window, boolean singleOutput) Returns a list of boundaries for multiple compaction output from minTimestamp to maxTimestamp.
private CompactionWindow	getIncomingWindow(long now)
private static long	getOldestToCompact(long maxAgeMillis, long now)
private String	getWindowStoragePolicy(long now, long windowStartMillis)
boolean	needsCompaction(Collection<HStoreFile> storeFiles, List<HStoreFile> filesCompacting) Heuristics for guessing whether we need minor compaction.
CompactionRequestImpl	selectMajorCompaction(ArrayList<HStoreFile> candidateSelection)
CompactionRequestImpl	selectMinorCompaction(ArrayList<HStoreFile> candidateSelection, boolean mayUseOffPeak, boolean mayBeStuck) We receive store files sorted in ascending order by seqId then scan the list of files.
boolean	shouldPerformMajorCompaction(Collection<HStoreFile> filesToCompact)

Methods inherited from class org.apache.hadoop.hbase.regionserver.compactions.SortedCompactionPolicy

checkMinFilesCriteria, filterBulk, getCurrentEligibleFiles, getNextMajorCompactTime, preSelectCompactionForCoprocessor, removeExcessFiles, selectCompaction, skipLargeFiles, throttleCompaction

Methods inherited from class org.apache.hadoop.hbase.regionserver.compactions.CompactionPolicy

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

private static final org.slf4j.Logger LOG

compactionPolicyPerWindow

private final RatioBasedCompactionPolicy compactionPolicyPerWindow

windowFactory

private final CompactionWindowFactory windowFactory

Constructor Detail

DateTieredCompactionPolicy

public DateTieredCompactionPolicy(org.apache.hadoop.conf.Configuration conf,
                                  StoreConfigInformation storeConfigInfo)
                           throws IOException

Throws:

IOException

Method Detail

needsCompaction

@InterfaceAudience.Private
public boolean needsCompaction(Collection<HStoreFile> storeFiles,
                                                          List<HStoreFile> filesCompacting)

Heuristics for guessing whether we need minor compaction.

Specified by:

needsCompaction in class SortedCompactionPolicy

shouldPerformMajorCompaction

public boolean shouldPerformMajorCompaction(Collection<HStoreFile> filesToCompact)
                                     throws IOException

Specified by:

shouldPerformMajorCompaction in class SortedCompactionPolicy

Parameters:

filesToCompact - Files to compact. Can be null.

Returns:

True if we should run a major compaction.

Throws:

IOException

createCompactionRequest

protected CompactionRequestImpl createCompactionRequest(ArrayList<HStoreFile> candidateSelection,
                                                        boolean tryingMajor,
                                                        boolean mayUseOffPeak,
                                                        boolean mayBeStuck)
                                                 throws IOException

Specified by:

createCompactionRequest in class SortedCompactionPolicy

Throws:

IOException

selectMajorCompaction

public CompactionRequestImpl selectMajorCompaction(ArrayList<HStoreFile> candidateSelection)

selectMinorCompaction

public CompactionRequestImpl selectMinorCompaction(ArrayList<HStoreFile> candidateSelection,
                                                   boolean mayUseOffPeak,
                                                   boolean mayBeStuck)
                                            throws IOException

We receive store files sorted in ascending order by seqId then scan the list of files. If the current file has a maxTimestamp older than last known maximum, treat this file as it carries the last known maximum. This way both seqId and timestamp are in the same order. If files carry the same maxTimestamps, they are ordered by seqId. We then reverse the list so they are ordered by seqId and maxTimestamp in descending order and build the time windows. All the out-of-order data into the same compaction windows, guaranteeing contiguous compaction based on sequence id.

Throws:

IOException

generateCompactionRequest

private DateTieredCompactionRequest generateCompactionRequest(ArrayList<HStoreFile> storeFiles,
                                                              CompactionWindow window,
                                                              boolean mayUseOffPeak,
                                                              boolean mayBeStuck,
                                                              int minThreshold,
                                                              long now)
                                                       throws IOException

Throws:

IOException

getCompactBoundariesForMajor

private List<Long> getCompactBoundariesForMajor(Collection<HStoreFile> filesToCompact,
                                                long now)

Return a list of boundaries for multiple compaction output in ascending order.

getCompactionBoundariesForMinor

private static List<Long> getCompactionBoundariesForMinor(CompactionWindow window,
                                                          boolean singleOutput)

Returns a list of boundaries for multiple compaction output from minTimestamp to maxTimestamp.

getIncomingWindow

private CompactionWindow getIncomingWindow(long now)

getOldestToCompact

private static long getOldestToCompact(long maxAgeMillis,
                                       long now)

getBoundariesStoragePolicyForMinor

private Map<Long,String> getBoundariesStoragePolicyForMinor(boolean singleOutput,
                                                            CompactionWindow window,
                                                            long now)

getBoundariesStoragePolicyForMajor

private Map<Long,String> getBoundariesStoragePolicyForMajor(List<Long> boundaries,
                                                            long now)

getWindowStoragePolicy

private String getWindowStoragePolicy(long now,
                                      long windowStartMillis)