org.apache.hadoop.hbase.regionserver

Class CompactionPipeline

java.lang.Object

org.apache.hadoop.hbase.regionserver.CompactionPipeline

@InterfaceAudience.Private
public class CompactionPipeline
extends Object

The compaction pipeline of a CompactingMemStore, is a FIFO queue of segments. It supports pushing a segment at the head of the pipeline and removing a segment from the tail when it is flushed to disk. It also supports swap method to allow the in-memory compaction swap a subset of the segments at the tail of the pipeline with a new (compacted) one. This swap succeeds only if the version number passed with the list of segments to swap is the same as the current version of the pipeline. Essentially, there are two methods which can change the structure of the pipeline: pushHead() and swap(), the later is used both by a flush to disk and by an in-memory compaction. The pipeline version is updated by swap(); it allows to identify conflicting operations at the suffix of the pipeline. The synchronization model is copy-on-write. Methods which change the structure of the pipeline (pushHead() and swap()) apply their changes in the context of a lock. They also make a read-only copy of the pipeline's list. Read methods read from a read-only copy. If a read method accesses the read-only copy more than once it makes a local copy of it to ensure it accesses the same copy. The methods getVersionedList(), getVersionedTail(), and flattenOneSegment() are also protected by a lock since they need to have a consistent (atomic) view of the pipeline list and version number.

Field Summary

Fields

Modifier and Type	Field and Description
static long	DEEP_OVERHEAD
static long	FIXED_OVERHEAD
private static org.slf4j.Logger	LOG
private LinkedList<ImmutableSegment>	pipeline
private LinkedList<ImmutableSegment>	readOnlyCopy
private RegionServicesForStores	region
private long	version Version is volatile to ensure it is atomically read when not using a lock.

Constructor Summary

Constructors

Constructor and Description
CompactionPipeline(RegionServicesForStores region)

Method Summary

Modifier and Type	Method and Description
private boolean	addFirst(ImmutableSegment segment)
boolean	flattenOneSegment(long requesterVersion, CompactingMemStore.IndexType idxType, MemStoreCompactionStrategy.Action action) If the caller holds the current version, go over the the pipeline and try to flatten each segment.
long	getMinSequenceId()
MemStoreSize	getPipelineSize()
List<? extends Segment>	getSegments()
private static int	getSegmentsCellsCount(List<? extends Segment> list)
private static long	getSegmentsHeapSize(List<? extends Segment> list)
private static long	getSegmentsKeySize(List<? extends Segment> list)
private static long	getSegmentsOffHeapSize(List<? extends Segment> list)
Segment	getTail()
MemStoreSize	getTailSize()
VersionedSegmentsList	getVersionedList()
VersionedSegmentsList	getVersionedTail()
boolean	isEmpty()
private void	matchAndRemoveSuffixFromPipeline(List<? extends Segment> suffix) Checking that the Segments in suffix input parameter is same as the Segments in pipeline one by one from the last element to the first element of suffix.
boolean	pushHead(MutableSegment segment)
private void	replaceAtIndex(int idx, ImmutableSegment newSegment)
long	size()
boolean	swap(VersionedSegmentsList versionedList, ImmutableSegment segment, boolean closeSuffix, boolean updateRegionSize) Swaps the versioned list at the tail of the pipeline with a new segment.
private void	swapSuffix(List<? extends Segment> suffix, ImmutableSegment segment, boolean closeSegmentsInSuffix) Must be called under the pipeline Lock.
private boolean	validateSuffixList(LinkedList<ImmutableSegment> suffix)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

private static final org.slf4j.Logger LOG

FIXED_OVERHEAD

public static final long FIXED_OVERHEAD

DEEP_OVERHEAD

public static final long DEEP_OVERHEAD

region

private final RegionServicesForStores region

pipeline

private final LinkedList<ImmutableSegment> pipeline

readOnlyCopy

private volatile LinkedList<ImmutableSegment> readOnlyCopy

version

private volatile long version

 Version is volatile to ensure it is atomically read when not using a lock.
 To indicate whether the suffix of pipeline changes:
 1.for pushHead(MutableSegment),new ImmutableSegment only
   added at Head, version not change.
 2.for swap(org.apache.hadoop.hbase.regionserver.VersionedSegmentsList, org.apache.hadoop.hbase.regionserver.ImmutableSegment, boolean, boolean),version increase.
 3.for replaceAtIndex(int, org.apache.hadoop.hbase.regionserver.ImmutableSegment),version increase.
 

Constructor Detail

CompactionPipeline

public CompactionPipeline(RegionServicesForStores region)

Method Detail

pushHead

public boolean pushHead(MutableSegment segment)

getVersionedList

public VersionedSegmentsList getVersionedList()

getVersionedTail

public VersionedSegmentsList getVersionedTail()

swap

public boolean swap(VersionedSegmentsList versionedList,
                    ImmutableSegment segment,
                    boolean closeSuffix,
                    boolean updateRegionSize)

Swaps the versioned list at the tail of the pipeline with a new segment. Swapping only if there were no changes to the suffix of the list since the version list was created.

Parameters:

versionedList - suffix of the pipeline to be replaced can be tail or all the pipeline

segment - new segment to replace the suffix. Can be null if the suffix just needs to be removed.

closeSuffix - whether to close the suffix (to release memory), as part of swapping it out During index merge op this will be false and for compaction it will be true.

updateRegionSize - whether to update the region size. Update the region size, when the pipeline is swapped as part of in-memory-flush and further merge/compaction. Don't update the region size when the swap is result of the snapshot (flush-to-disk).

Returns:

true iff swapped tail with new segment

getSegmentsHeapSize

private static long getSegmentsHeapSize(List<? extends Segment> list)

getSegmentsOffHeapSize

private static long getSegmentsOffHeapSize(List<? extends Segment> list)

getSegmentsKeySize

private static long getSegmentsKeySize(List<? extends Segment> list)

getSegmentsCellsCount

private static int getSegmentsCellsCount(List<? extends Segment> list)

flattenOneSegment

public boolean flattenOneSegment(long requesterVersion,
                                 CompactingMemStore.IndexType idxType,
                                 MemStoreCompactionStrategy.Action action)

If the caller holds the current version, go over the the pipeline and try to flatten each segment. Flattening is replacing the ConcurrentSkipListMap based CellSet to CellArrayMap based. Flattening of the segment that initially is not based on ConcurrentSkipListMap has no effect. Return after one segment is successfully flatten.

Returns:

true iff a segment was successfully flattened

isEmpty

public boolean isEmpty()

getSegments

public List<? extends Segment> getSegments()

size

public long size()

getMinSequenceId

public long getMinSequenceId()

getTailSize

public MemStoreSize getTailSize()

getPipelineSize

public MemStoreSize getPipelineSize()

swapSuffix

private void swapSuffix(List<? extends Segment> suffix,
                        ImmutableSegment segment,
                        boolean closeSegmentsInSuffix)

Must be called under the pipeline Lock.

matchAndRemoveSuffixFromPipeline

private void matchAndRemoveSuffixFromPipeline(List<? extends Segment> suffix)

Checking that the Segments in suffix input parameter is same as the Segments in pipeline one by one from the last element to the first element of suffix. If matched, remove suffix from pipeline.
Must be called under the pipeline Lock.

replaceAtIndex

private void replaceAtIndex(int idx,
                            ImmutableSegment newSegment)

getTail

public Segment getTail()

addFirst

private boolean addFirst(ImmutableSegment segment)

validateSuffixList

private boolean validateSuffixList(LinkedList<ImmutableSegment> suffix)