Field Detail

• LOG

  private static final org.apache.commons.logging.Log LOG

• random

  private final Random random

  Used calculation jitter

Constructor Detail

• RatioBasedCompactionPolicy

  public RatioBasedCompactionPolicy(org.apache.hadoop.conf.Configuration conf,
                            StoreConfigInformation storeConfigInfo)

Method Detail

• getCurrentEligibleFiles

  private ArrayList<StoreFile> getCurrentEligibleFiles(ArrayList<StoreFile> candidateFiles,
                                             List<StoreFile> filesCompacting)

• preSelectCompactionForCoprocessor

  public List<StoreFile> preSelectCompactionForCoprocessor(Collection<StoreFile> candidates,
                                                  List<StoreFile> filesCompacting)

• selectCompaction

  public CompactionRequest selectCompaction(Collection<StoreFile> candidateFiles,
                                   List<StoreFile> filesCompacting,
                                   boolean isUserCompaction,
                                   boolean mayUseOffPeak,
                                   boolean forceMajor)
                                     throws IOException

  Parameters:

  candidateFiles - candidate files, ordered from oldest to newest. All files in store.

  Returns:

  subset copy of candidate list that meets compaction criteria

  Throws:

  IOException

• skipLargeFiles

  private ArrayList<StoreFile> skipLargeFiles(ArrayList<StoreFile> candidates,
                                    boolean mayUseOffpeak)

  Parameters:

  candidates - pre-filtrate

  Returns:

  filtered subset exclude all files above maxCompactSize Also save all references. We MUST compact them

• filterBulk

  private ArrayList<StoreFile> filterBulk(ArrayList<StoreFile> candidates)

  Parameters:

  candidates - pre-filtrate

  Returns:

  filtered subset exclude all bulk load files if configured

• removeExcessFiles

  private ArrayList<StoreFile> removeExcessFiles(ArrayList<StoreFile> candidates,
                                       boolean isUserCompaction,
                                       boolean isMajorCompaction)

  Parameters:

  candidates - pre-filtrate

  Returns:

  filtered subset take upto maxFilesToCompact from the start

• checkMinFilesCriteria

  private ArrayList<StoreFile> checkMinFilesCriteria(ArrayList<StoreFile> candidates)

  Parameters:

  candidates - pre-filtrate

  Returns:

  filtered subset forget the compactionSelection if we don't have enough files

• applyCompactionPolicy

  ArrayList<StoreFile> applyCompactionPolicy(ArrayList<StoreFile> candidates,
                                           boolean mayUseOffPeak,
                                           boolean mayBeStuck)
                                       throws IOException

  Parameters:

  candidates - pre-filtrate

  Returns:

  filtered subset -- Default minor compaction selection algorithm: choose CompactSelection from candidates -- First exclude bulk-load files if indicated in configuration. Start at the oldest file and stop when you find the first file that meets compaction criteria: (1) a recently-flushed, small file (i.e. <= minCompactSize) OR (2) within the compactRatio of sum(newer_files) Given normal skew, any newer files will also meet this criteria

  Additional Note: If fileSizes.size() >> maxFilesToCompact, we will recurse on compact(). Consider the oldest files first to avoid a situation where we always compact [end-threshold,end). Then, the last file becomes an aggregate of the previous compactions. normal skew: older ----> newer (increasing seqID) _ | | _ | | | | _ --|-|- |-|- |-|---_-------_------- minCompactSize | | | | | | | | _ | | | | | | | | | | | | | | | | | | | | | | | | | |

  Throws:

  IOException

• isMajorCompaction

  public boolean isMajorCompaction(Collection<StoreFile> filesToCompact)
                            throws IOException

  Specified by:

  isMajorCompaction in class CompactionPolicy

  Parameters:

  filesToCompact - Files to compact. Can be null.

  Returns:

  True if we should run a major compaction.

  Throws:

  IOException

• getNextMajorCompactTime

  public long getNextMajorCompactTime(Collection<StoreFile> filesToCompact)

  Parameters:

  filesToCompact -

  Returns:

  When to run next major compaction

• throttleCompaction

  public boolean throttleCompaction(long compactionSize)

  Specified by:

  throttleCompaction in class CompactionPolicy

  Parameters:

  compactionSize - Total size of some compaction

  Returns:

  whether this should be a large or small compaction

• needsCompaction

  public boolean needsCompaction(Collection<StoreFile> storeFiles,
                        List<StoreFile> filesCompacting)