Class Compactor<T extends CellSink>
java.lang.Object
org.apache.hadoop.hbase.regionserver.compactions.Compactor<T>
- Direct Known Subclasses:
AbstractMultiOutputCompactor
,DefaultCompactor
A compactor is a compaction algorithm associated a given policy. Base class also contains
reusable parts for implementing compactors (what is common and what isn't is evolving).
Compactions might be concurrent against a given store and the Compactor is shared among them. Do not put mutable state into class fields. All Compactor class fields should be final or effectively final. 'keepSeqIdPeriod' is an exception to this rule because unit tests may set it.
-
Nested Class Summary
Modifier and TypeClassDescriptionprotected static interface
protected static class
The sole reason this class exists is that java has no ref/out/pointer parameters.protected static interface
-
Field Summary
Modifier and TypeFieldDescriptionprotected static final long
protected final int
protected final long
protected final org.apache.hadoop.conf.Configuration
protected final Compactor.InternalScannerFactory
protected final boolean
protected final boolean
protected int
specify how many days to keep MVCC values during major compactionprivate static final org.slf4j.Logger
protected static final String
protected final Compression.Algorithm
protected static final String
protected final Compression.Algorithm
private final Set<CompactionProgress>
protected final HStore
-
Constructor Summary
-
Method Summary
Modifier and TypeMethodDescriptionprotected abstract void
abortWriter
(T writer) protected abstract List<org.apache.hadoop.fs.Path>
commitWriter
(T writer, Compactor.FileDetails fd, CompactionRequestImpl request) protected final List<org.apache.hadoop.fs.Path>
compact
(CompactionRequestImpl request, Compactor.InternalScannerFactory scannerFactory, Compactor.CellSinkFactory<T> sinkFactory, ThroughputController throughputController, User user) private List<StoreFileScanner>
createFileScanners
(Collection<HStoreFile> filesToCompact, long smallestReadPoint, boolean useDropBehind) Creates file scanners for compaction.protected final CreateStoreFileWriterParams
createParams
(Compactor.FileDetails fd, boolean shouldDropBehind, boolean major, Consumer<org.apache.hadoop.fs.Path> writerCreationTracker) protected InternalScanner
createScanner
(HStore store, ScanInfo scanInfo, List<StoreFileScanner> scanners, long smallestReadPoint, long earliestPutTs, byte[] dropDeletesFromRow, byte[] dropDeletesToRow) protected InternalScanner
createScanner
(HStore store, ScanInfo scanInfo, List<StoreFileScanner> scanners, ScanType scanType, long smallestReadPoint, long earliestPutTs) protected final StoreFileWriter
createWriter
(Compactor.FileDetails fd, boolean shouldDropBehind, boolean major, Consumer<org.apache.hadoop.fs.Path> writerCreationTracker) Creates a writer for a new file.protected final StoreFileWriter
createWriter
(Compactor.FileDetails fd, boolean shouldDropBehind, String fileStoragePolicy, boolean major, Consumer<org.apache.hadoop.fs.Path> writerCreationTracker) private Compactor.FileDetails
getFileDetails
(Collection<HStoreFile> filesToCompact, boolean allFiles, boolean major) Extracts some details about the files to compact that are commonly needed by compactors.Return the aggregate progress for all currently active compactions.private long
boolean
protected boolean
performCompaction
(Compactor.FileDetails fd, InternalScanner scanner, CellSink writer, long smallestReadPoint, boolean cleanSeqId, ThroughputController throughputController, CompactionRequestImpl request, CompactionProgress progress) Performs the compaction.private InternalScanner
postCompactScannerOpen
(CompactionRequestImpl request, ScanType scanType, InternalScanner scanner, User user) Calls coprocessor, if any, to create scanners - after normal scanner creation.private ScanInfo
preCompactScannerOpen
(CompactionRequestImpl request, ScanType scanType, User user)
-
Field Details
-
LOG
-
COMPACTION_PROGRESS_LOG_INTERVAL
- See Also:
-
conf
-
store
-
compactionKVMax
-
compactScannerSizeLimit
-
majorCompactionCompression
-
minorCompactionCompression
-
keepSeqIdPeriod
specify how many days to keep MVCC values during major compaction -
MAJOR_COMPACTION_DROP_CACHE
- See Also:
-
MINOR_COMPACTION_DROP_CACHE
- See Also:
-
dropCacheMajor
-
dropCacheMinor
-
progressSet
-
defaultScannerFactory
-
-
Constructor Details
-
Compactor
-
-
Method Details
-
getFileDetails
private Compactor.FileDetails getFileDetails(Collection<HStoreFile> filesToCompact, boolean allFiles, boolean major) throws IOException Extracts some details about the files to compact that are commonly needed by compactors.- Parameters:
filesToCompact
- Files.allFiles
- Whether all files are included for compaction- Returns:
- The result.
- Throws:
IOException
-
createFileScanners
private List<StoreFileScanner> createFileScanners(Collection<HStoreFile> filesToCompact, long smallestReadPoint, boolean useDropBehind) throws IOException Creates file scanners for compaction.- Parameters:
filesToCompact
- Files.- Returns:
- Scanners.
- Throws:
IOException
-
getSmallestReadPoint
-
createParams
protected final CreateStoreFileWriterParams createParams(Compactor.FileDetails fd, boolean shouldDropBehind, boolean major, Consumer<org.apache.hadoop.fs.Path> writerCreationTracker) -
createWriter
protected final StoreFileWriter createWriter(Compactor.FileDetails fd, boolean shouldDropBehind, boolean major, Consumer<org.apache.hadoop.fs.Path> writerCreationTracker) throws IOException Creates a writer for a new file.- Parameters:
fd
- The file details.- Returns:
- Writer for a new StoreFile
- Throws:
IOException
- if creation failed
-
createWriter
protected final StoreFileWriter createWriter(Compactor.FileDetails fd, boolean shouldDropBehind, String fileStoragePolicy, boolean major, Consumer<org.apache.hadoop.fs.Path> writerCreationTracker) throws IOException - Throws:
IOException
-
preCompactScannerOpen
private ScanInfo preCompactScannerOpen(CompactionRequestImpl request, ScanType scanType, User user) throws IOException - Throws:
IOException
-
postCompactScannerOpen
private InternalScanner postCompactScannerOpen(CompactionRequestImpl request, ScanType scanType, InternalScanner scanner, User user) throws IOException Calls coprocessor, if any, to create scanners - after normal scanner creation.- Parameters:
request
- Compaction request.scanType
- Scan type.scanner
- The default scanner created for compaction.- Returns:
- Scanner scanner to use (usually the default); null if compaction should not proceed.
- Throws:
IOException
-
compact
protected final List<org.apache.hadoop.fs.Path> compact(CompactionRequestImpl request, Compactor.InternalScannerFactory scannerFactory, Compactor.CellSinkFactory<T> sinkFactory, ThroughputController throughputController, User user) throws IOException - Throws:
IOException
-
commitWriter
protected abstract List<org.apache.hadoop.fs.Path> commitWriter(T writer, Compactor.FileDetails fd, CompactionRequestImpl request) throws IOException - Throws:
IOException
-
abortWriter
- Throws:
IOException
-
performCompaction
protected boolean performCompaction(Compactor.FileDetails fd, InternalScanner scanner, CellSink writer, long smallestReadPoint, boolean cleanSeqId, ThroughputController throughputController, CompactionRequestImpl request, CompactionProgress progress) throws IOException Performs the compaction.- Parameters:
fd
- FileDetails of cell sink writerscanner
- Where to read from.writer
- Where to write to.smallestReadPoint
- Smallest read point.cleanSeqId
- When true, remove seqId(used to be mvcc) value which is <= smallestReadPointrequest
- compaction request.progress
- Progress reporter.- Returns:
- Whether compaction ended; false if it was interrupted for some reason.
- Throws:
IOException
-
createScanner
protected InternalScanner createScanner(HStore store, ScanInfo scanInfo, List<StoreFileScanner> scanners, ScanType scanType, long smallestReadPoint, long earliestPutTs) throws IOException - Parameters:
store
- storescanners
- Store file scanners.scanType
- Scan type.smallestReadPoint
- Smallest MVCC read point.earliestPutTs
- Earliest put across all files.- Returns:
- A compaction scanner.
- Throws:
IOException
-
createScanner
protected InternalScanner createScanner(HStore store, ScanInfo scanInfo, List<StoreFileScanner> scanners, long smallestReadPoint, long earliestPutTs, byte[] dropDeletesFromRow, byte[] dropDeletesToRow) throws IOException - Parameters:
store
- The store.scanners
- Store file scanners.smallestReadPoint
- Smallest MVCC read point.earliestPutTs
- Earliest put across all files.dropDeletesFromRow
- Drop deletes starting with this row, inclusive. Can be null.dropDeletesToRow
- Drop deletes ending with this row, exclusive. Can be null.- Returns:
- A compaction scanner.
- Throws:
IOException
-
getProgress
Return the aggregate progress for all currently active compactions. -
isCompacting
-