Class CompactionPipeline
java.lang.Object
org.apache.hadoop.hbase.regionserver.CompactionPipeline
The compaction pipeline of a
CompactingMemStore
, is a FIFO queue of segments. It supports
pushing a segment at the head of the pipeline and removing a segment from the tail when it is
flushed to disk. It also supports swap method to allow the in-memory compaction swap a subset of
the segments at the tail of the pipeline with a new (compacted) one. This swap succeeds only if
the version number passed with the list of segments to swap is the same as the current version of
the pipeline. Essentially, there are two methods which can change the structure of the pipeline:
pushHead() and swap(), the later is used both by a flush to disk and by an in-memory compaction.
The pipeline version is updated by swap(); it allows to identify conflicting operations at the
suffix of the pipeline. The synchronization model is copy-on-write. Methods which change the
structure of the pipeline (pushHead(), flattenOneSegment() and swap()) apply their changes in the
context of a lock. They also make a read-only copy of the pipeline's list. Read methods read from
a read-only copy. If a read method accesses the read-only copy more than once it makes a local
copy of it to ensure it accesses the same copy. The methods getVersionedList(),
getVersionedTail(), and flattenOneSegment() are also protected by a lock since they need to have
a consistent (atomic) view of the pipeline list and version number.-
Field Summary
Modifier and TypeFieldDescriptionstatic final long
static final long
private static final org.slf4j.Logger
private final LinkedList<ImmutableSegment>
private LinkedList<ImmutableSegment>
private final RegionServicesForStores
private long
Version is volatile to ensure it is atomically read when not using a lock. -
Constructor Summary
-
Method Summary
Modifier and TypeMethodDescriptionprivate boolean
addFirst
(ImmutableSegment segment) boolean
flattenOneSegment
(long requesterVersion, CompactingMemStore.IndexType idxType, MemStoreCompactionStrategy.Action action) If the caller holds the current version, go over the the pipeline and try to flatten each segment.long
private static int
getSegmentsCellsCount
(List<? extends Segment> list) private static long
getSegmentsHeapSize
(List<? extends Segment> list) private static long
getSegmentsKeySize
(List<? extends Segment> list) private static long
getSegmentsOffHeapSize
(List<? extends Segment> list) getTail()
boolean
isEmpty()
private void
matchAndRemoveSuffixFromPipeline
(List<? extends Segment> suffix) boolean
pushHead
(MutableSegment segment) private void
replaceAtIndex
(int idx, ImmutableSegment newSegment) long
size()
boolean
swap
(VersionedSegmentsList versionedList, ImmutableSegment segment, boolean closeSuffix, boolean updateRegionSize) Swaps the versioned list at the tail of the pipeline with a new segment.private void
swapSuffix
(List<? extends Segment> suffix, ImmutableSegment segment, boolean closeSegmentsInSuffix) Must be called under thepipeline
Lock.private boolean
validateSuffixList
(LinkedList<ImmutableSegment> suffix)
-
Field Details
-
LOG
-
FIXED_OVERHEAD
-
DEEP_OVERHEAD
-
region
-
pipeline
-
readOnlyCopy
-
version
Version is volatile to ensure it is atomically read when not using a lock. To indicate whether the suffix of pipeline changes: 1.for
pushHead(MutableSegment)
,newImmutableSegment
only added at Head,version
not change. 2.forswap(org.apache.hadoop.hbase.regionserver.VersionedSegmentsList, org.apache.hadoop.hbase.regionserver.ImmutableSegment, boolean, boolean)
,version
increase. 3.forreplaceAtIndex(int, org.apache.hadoop.hbase.regionserver.ImmutableSegment)
,version
increase.
-
-
Constructor Details
-
CompactionPipeline
-
-
Method Details
-
pushHead
-
getVersionedList
-
getVersionedTail
-
swap
public boolean swap(VersionedSegmentsList versionedList, ImmutableSegment segment, boolean closeSuffix, boolean updateRegionSize) Swaps the versioned list at the tail of the pipeline with a new segment. Swapping only if there were no changes to the suffix of the list since the version list was created.- Parameters:
versionedList
- suffix of the pipeline to be replaced can be tail or all the pipelinesegment
- new segment to replace the suffix. Can be null if the suffix just needs to be removed.closeSuffix
- whether to close the suffix (to release memory), as part of swapping it out During index merge op this will be false and for compaction it will be true.updateRegionSize
- whether to update the region size. Update the region size, when the pipeline is swapped as part of in-memory-flush and further merge/compaction. Don't update the region size when the swap is result of the snapshot (flush-to-disk).- Returns:
- true iff swapped tail with new segment
-
getSegmentsHeapSize
-
getSegmentsOffHeapSize
-
getSegmentsKeySize
-
getSegmentsCellsCount
-
flattenOneSegment
public boolean flattenOneSegment(long requesterVersion, CompactingMemStore.IndexType idxType, MemStoreCompactionStrategy.Action action) If the caller holds the current version, go over the the pipeline and try to flatten each segment. Flattening is replacing the ConcurrentSkipListMap based CellSet to CellArrayMap based. Flattening of the segment that initially is not based on ConcurrentSkipListMap has no effect. Return after one segment is successfully flatten.- Returns:
- true iff a segment was successfully flattened
-
isEmpty
-
getSegments
-
size
-
getMinSequenceId
-
getTailSize
-
getPipelineSize
-
swapSuffix
private void swapSuffix(List<? extends Segment> suffix, ImmutableSegment segment, boolean closeSegmentsInSuffix) Must be called under thepipeline
Lock. -
matchAndRemoveSuffixFromPipeline
-
replaceAtIndex
-
getTail
-
addFirst
-
validateSuffixList
-