Class DateTieredCompactionPolicy
java.lang.Object
org.apache.hadoop.hbase.regionserver.compactions.CompactionPolicy
org.apache.hadoop.hbase.regionserver.compactions.SortedCompactionPolicy
org.apache.hadoop.hbase.regionserver.compactions.DateTieredCompactionPolicy
@LimitedPrivate("Configuration")
public class DateTieredCompactionPolicy
extends SortedCompactionPolicy
HBASE-15181 This is a simple implementation of date-based tiered compaction similar to
Cassandra's for the following benefits:
- Improve date-range-based scan by structuring store files in date-based tiered layout.
- Reduce compaction overhead.
- Improve TTL efficiency.
- has mostly date-based data write and scan and a focus on the most recent data.
-
Field Summary
Modifier and TypeFieldDescriptionprivate final RatioBasedCompactionPolicy
private static final org.slf4j.Logger
private final CompactionWindowFactory
Fields inherited from class org.apache.hadoop.hbase.regionserver.compactions.CompactionPolicy
comConf, storeConfigInfo
-
Constructor Summary
ConstructorDescriptionDateTieredCompactionPolicy
(org.apache.hadoop.conf.Configuration conf, StoreConfigInformation storeConfigInfo) -
Method Summary
Modifier and TypeMethodDescriptionprotected CompactionRequestImpl
createCompactionRequest
(ArrayList<HStoreFile> candidateSelection, boolean tryingMajor, boolean mayUseOffPeak, boolean mayBeStuck) private DateTieredCompactionRequest
generateCompactionRequest
(ArrayList<HStoreFile> storeFiles, CompactionWindow window, boolean mayUseOffPeak, boolean mayBeStuck, int minThreshold, long now) getBoundariesStoragePolicyForMajor
(List<Long> boundaries, long now) getBoundariesStoragePolicyForMinor
(boolean singleOutput, CompactionWindow window, long now) getCompactBoundariesForMajor
(Collection<HStoreFile> filesToCompact, long now) Return a list of boundaries for multiple compaction output in ascending order.getCompactionBoundariesForMinor
(CompactionWindow window, boolean singleOutput) Returns a list of boundaries for multiple compaction output from minTimestamp to maxTimestamp.private CompactionWindow
getIncomingWindow
(long now) private static long
getOldestToCompact
(long maxAgeMillis, long now) private String
getWindowStoragePolicy
(long now, long windowStartMillis) boolean
needsCompaction
(Collection<HStoreFile> storeFiles, List<HStoreFile> filesCompacting) Heuristics for guessing whether we need minor compaction.selectMajorCompaction
(ArrayList<HStoreFile> candidateSelection) selectMinorCompaction
(ArrayList<HStoreFile> candidateSelection, boolean mayUseOffPeak, boolean mayBeStuck) We receive store files sorted in ascending order by seqId then scan the list of files.boolean
shouldPerformMajorCompaction
(Collection<HStoreFile> filesToCompact) Methods inherited from class org.apache.hadoop.hbase.regionserver.compactions.SortedCompactionPolicy
checkMinFilesCriteria, filterBulk, getCurrentEligibleFiles, getNextMajorCompactTime, preSelectCompactionForCoprocessor, removeExcessFiles, selectCompaction, skipLargeFiles, throttleCompaction
Methods inherited from class org.apache.hadoop.hbase.regionserver.compactions.CompactionPolicy
getConf, setConf
-
Field Details
-
LOG
-
compactionPolicyPerWindow
-
windowFactory
-
-
Constructor Details
-
DateTieredCompactionPolicy
public DateTieredCompactionPolicy(org.apache.hadoop.conf.Configuration conf, StoreConfigInformation storeConfigInfo) throws IOException - Throws:
IOException
-
-
Method Details
-
needsCompaction
@Private public boolean needsCompaction(Collection<HStoreFile> storeFiles, List<HStoreFile> filesCompacting) Heuristics for guessing whether we need minor compaction.- Specified by:
needsCompaction
in classSortedCompactionPolicy
-
shouldPerformMajorCompaction
public boolean shouldPerformMajorCompaction(Collection<HStoreFile> filesToCompact) throws IOException - Specified by:
shouldPerformMajorCompaction
in classSortedCompactionPolicy
- Parameters:
filesToCompact
- Files to compact. Can be null.- Returns:
- True if we should run a major compaction.
- Throws:
IOException
-
createCompactionRequest
protected CompactionRequestImpl createCompactionRequest(ArrayList<HStoreFile> candidateSelection, boolean tryingMajor, boolean mayUseOffPeak, boolean mayBeStuck) throws IOException - Specified by:
createCompactionRequest
in classSortedCompactionPolicy
- Throws:
IOException
-
selectMajorCompaction
-
selectMinorCompaction
public CompactionRequestImpl selectMinorCompaction(ArrayList<HStoreFile> candidateSelection, boolean mayUseOffPeak, boolean mayBeStuck) throws IOException We receive store files sorted in ascending order by seqId then scan the list of files. If the current file has a maxTimestamp older than last known maximum, treat this file as it carries the last known maximum. This way both seqId and timestamp are in the same order. If files carry the same maxTimestamps, they are ordered by seqId. We then reverse the list so they are ordered by seqId and maxTimestamp in descending order and build the time windows. All the out-of-order data into the same compaction windows, guaranteeing contiguous compaction based on sequence id.- Throws:
IOException
-
generateCompactionRequest
private DateTieredCompactionRequest generateCompactionRequest(ArrayList<HStoreFile> storeFiles, CompactionWindow window, boolean mayUseOffPeak, boolean mayBeStuck, int minThreshold, long now) throws IOException - Throws:
IOException
-
getCompactBoundariesForMajor
Return a list of boundaries for multiple compaction output in ascending order. -
getCompactionBoundariesForMinor
private static List<Long> getCompactionBoundariesForMinor(CompactionWindow window, boolean singleOutput) Returns a list of boundaries for multiple compaction output from minTimestamp to maxTimestamp. -
getIncomingWindow
-
getOldestToCompact
-
getBoundariesStoragePolicyForMinor
private Map<Long,String> getBoundariesStoragePolicyForMinor(boolean singleOutput, CompactionWindow window, long now) -
getBoundariesStoragePolicyForMajor
-
getWindowStoragePolicy
-