StripeStoreFileManager (Apache HBase 2.2.3 API)

java.lang.Object
- org.apache.hadoop.hbase.regionserver.StripeStoreFileManager

All Implemented Interfaces:

StripeCompactionPolicy.StripeInformationProvider, StoreFileManager
```
@InterfaceAudience.Private
public class StripeStoreFileManager
extends Object
implements StoreFileManager, StripeCompactionPolicy.StripeInformationProvider
```
Stripe implementation of StoreFileManager. Not thread safe - relies on external locking (in HStore). Collections that this class returns are immutable or unique to the call, so they should be safe. Stripe store splits the key space of the region into non-overlapping stripes, as well as some recent files that have all the keys (level 0). Each stripe contains a set of files. When L0 is compacted, it's split into the files corresponding to existing stripe boundaries, that can thus be added to stripes. When scan or get happens, it only has to read the files from the corresponding stripes. See StripeCompationPolicy on how the stripes are determined; this class doesn't care. This class should work together with StripeCompactionPolicy and StripeCompactor. With regard to how they work, we make at least the following (reasonable) assumptions: - Compaction produces one file per new stripe (if any); that is easy to change. - Compaction has one contiguous set of stripes both in and out, except if L0 is involved.

Nested Class Summary

Nested Classes
Modifier and Type	Class and Description
`private class`	`StripeStoreFileManager.CompactionOrFlushMergeCopy` Non-static helper class for merging compaction or flush results.
`private static class`	`StripeStoreFileManager.KeyBeforeConcatenatedLists` An extension of ConcatenatedLists that has several peculiar properties.
`private static class`	`StripeStoreFileManager.State` The state class.

Field Summary

Fields
Modifier and Type	Field and Description
`private int`	`blockingFileCount`
`private CellComparator`	`cellComparator`
`private StripeStoreConfig`	`config`
`private HashMap<HStoreFile,byte[]>`	`fileEnds`
`private HashMap<HStoreFile,byte[]>`	`fileStarts` Cached file metadata (or overrides as the case may be)
`(package private) static byte[]`	`INVALID_KEY`
`private static byte[]`	`INVALID_KEY_IN_MAP` Normally invalid key is null, but in the map null is the result for "no key"; so use the following constant value in these maps instead.
`private static org.slf4j.Logger`	`LOG`
`private static Bytes.RowEndKeyComparator`	`MAP_COMPARATOR`
`static byte[]`	`OPEN_KEY` The key value used for range boundary, indicating that the boundary is open (i.e.
`private StripeStoreFileManager.State`	`state`
`static byte[]`	`STRIPE_END_KEY`
`static byte[]`	`STRIPE_START_KEY` The file metadata fields that contain the stripe information.

Constructor Summary

Constructors
Constructor and Description
`StripeStoreFileManager(CellComparator kvComparator, org.apache.hadoop.conf.Configuration conf, StripeStoreConfig config)`

Method Summary

All Methods Static Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`void`	`addCompactionResults(Collection<HStoreFile> compactedFiles, Collection<HStoreFile> results)` Adds only the new compaction results into the structure.
`org.apache.hbase.thirdparty.com.google.common.collect.ImmutableCollection<HStoreFile>`	`clearCompactedFiles()` Clears all the compacted files and returns them.
`org.apache.hbase.thirdparty.com.google.common.collect.ImmutableCollection<HStoreFile>`	`clearFiles()` Clears all the files currently in use and returns them.
`private void`	`debugDumpState(String string)`
`private byte[]`	`endOf(HStoreFile sf)`
`private void`	`ensureEdgeStripeMetadata(ArrayList<HStoreFile> stripe, boolean isFirst)`
`private void`	`ensureLevel0Metadata(HStoreFile sf)`
`private Collection<HStoreFile>`	`findExpiredFiles(org.apache.hbase.thirdparty.com.google.common.collect.ImmutableList<HStoreFile> stripe, long maxTs, List<HStoreFile> filesCompacting, Collection<HStoreFile> expiredStoreFiles)`
`private int`	`findStripeForRow(byte[] row, boolean isStart)` Finds the stripe index for the stripe containing a row provided externally for get/scan.
`private int`	`findStripeIndexByEndRow(byte[] endRow)` Finds the stripe index by end row.
`Iterator<HStoreFile>`	`getCandidateFilesForRowKeyBefore(KeyValue targetKey)` See `StoreFileManager.getCandidateFilesForRowKeyBefore(KeyValue)` for details on this methods.
`Collection<HStoreFile>`	`getCompactedfiles()` List of compacted files inside this store that needs to be excluded in reads because further new reads will be using only the newly created files out of compaction.
`int`	`getCompactedFilesCount()` Returns the number of compacted files.
`double`	`getCompactionPressure()`
`byte[]`	`getEndRow(int stripeIndex)` Gets the end row for a given stripe.
`Collection<HStoreFile>`	`getFilesForScan(byte[] startRow, boolean includeStartRow, byte[] stopRow, boolean includeStopRow)` Gets the store files to scan for a Scan or Get request.
`List<HStoreFile>`	`getLevel0Files()`
`private double`	`getMidStripeSplitRatio(long smallerSize, long largerSize, long lastLargerSize)`
`Optional<byte[]>`	`getSplitPoint()` Override of getSplitPoint that determines the split point as the boundary between two stripes, unless it causes significant imbalance between split sides' sizes.
`private Optional<byte[]>`	`getSplitPointFromAllFiles()`
`byte[]`	`getStartRow(int stripeIndex)` Gets the start row for a given stripe.
`int`	`getStoreCompactionPriority()`
`Comparator<HStoreFile>`	`getStoreFileComparator()`
`int`	`getStorefileCount()` Returns the number of files currently in use.
`Collection<HStoreFile>`	`getStorefiles()` Gets the snapshot of the store files currently in use.
`List<byte[]>`	`getStripeBoundaries()`
`int`	`getStripeCount()`
`private long`	`getStripeFilesSize(int stripeIndex)` Gets the total size of all files in the stripe.
`ArrayList<org.apache.hbase.thirdparty.com.google.common.collect.ImmutableList<HStoreFile>>`	`getStripes()`
`Collection<HStoreFile>`	`getUnneededFiles(long maxTs, List<HStoreFile> filesCompacting)`
`private static void`	`insertFileIntoStripe(ArrayList<HStoreFile> stripe, HStoreFile sf)` Inserts a file in the correct place (by seqnum) in a stripe copy.
`void`	`insertNewFiles(Collection<HStoreFile> sfs)` Adds new files, either for from MemStore flush or bulk insert, into the structure.
`private static boolean`	`isInvalid(byte[] key)` Checks whether the key is invalid (e.g.
`private static boolean`	`isOpen(byte[] key)` Checks whether the key indicates an open interval boundary (i.e.
`private static boolean`	`isOpen(Cell key)`
`void`	`loadFiles(List<HStoreFile> storeFiles)` Loads the initial store files into empty StoreFileManager.
`private void`	`loadUnclassifiedStoreFiles(List<HStoreFile> storeFiles)` Loads initial store files that were picked up from some physical location pertaining to this store (presumably).
`private void`	`markCompactedAway(Collection<HStoreFile> compactedFiles)`
`private int`	`nonOpenRowCompare(byte[] k1, byte[] k2)` Compare two keys.
`private int`	`nonOpenRowCompare(Cell k1, byte[] k2)`
`void`	`removeCompactedFiles(Collection<HStoreFile> compactedFiles)` Remove the compacted files
`private boolean`	`rowEquals(byte[] k1, byte[] k2)` Compare two keys for equality.
`private byte[]`	`startOf(HStoreFile sf)`
`Iterator<HStoreFile>`	`updateCandidateFilesForRowKeyBefore(Iterator<HStoreFile> candidateFiles, KeyValue targetKey, Cell candidate)` See `StoreFileManager.getCandidateFilesForRowKeyBefore(KeyValue)` and `StoreFileManager.updateCandidateFilesForRowKeyBefore(Iterator, KeyValue, Cell)` for details on this methods.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - LOG
```
private static final org.slf4j.Logger LOG
```
  - STRIPE_START_KEY
```
public static final byte[] STRIPE_START_KEY
```
    The file metadata fields that contain the stripe information.
  - STRIPE_END_KEY
```
public static final byte[] STRIPE_END_KEY
```
  - MAP_COMPARATOR
```
private static final Bytes.RowEndKeyComparator MAP_COMPARATOR
```
  - OPEN_KEY
```
public static final byte[] OPEN_KEY
```
    The key value used for range boundary, indicating that the boundary is open (i.e. +-inf).
  - INVALID_KEY
```
static final byte[] INVALID_KEY
```
  - state
```
private StripeStoreFileManager.State state
```
  - fileStarts
```
private HashMap<HStoreFile,byte[]> fileStarts
```
    Cached file metadata (or overrides as the case may be)
  - fileEnds
```
private HashMap<HStoreFile,byte[]> fileEnds
```
  - INVALID_KEY_IN_MAP
```
private static final byte[] INVALID_KEY_IN_MAP
```
    Normally invalid key is null, but in the map null is the result for "no key"; so use the following constant value in these maps instead. Note that this is a constant and we use it to compare by reference when we read from the map.
  - cellComparator
```
private final CellComparator cellComparator
```
  - config
```
private StripeStoreConfig config
```
  - blockingFileCount
```
private final int blockingFileCount
```
- Constructor Detail
  - StripeStoreFileManager
```
public StripeStoreFileManager(CellComparator kvComparator,
                              org.apache.hadoop.conf.Configuration conf,
                              StripeStoreConfig config)
```
- Method Detail
  - loadFiles
```
public void loadFiles(List<HStoreFile> storeFiles)
```
    Description copied from interface: StoreFileManager
    
    Loads the initial store files into empty StoreFileManager.
    
    Specified by:
    
    loadFiles in interface StoreFileManager
    
    Parameters:
    
    storeFiles - The files to load.
  - getStorefiles
```
public Collection<HStoreFile> getStorefiles()
```
    Description copied from interface: StoreFileManager
    
    Gets the snapshot of the store files currently in use. Can be used for things like metrics and checks; should not assume anything about relations between store files in the list.
    
    Specified by:
    
    getStorefiles in interface StripeCompactionPolicy.StripeInformationProvider
    
    Specified by:
    
    getStorefiles in interface StoreFileManager
    
    Returns:
    
    The list of StoreFiles.
  - getCompactedfiles
```
public Collection<HStoreFile> getCompactedfiles()
```
    Description copied from interface: StoreFileManager
    
    List of compacted files inside this store that needs to be excluded in reads because further new reads will be using only the newly created files out of compaction. These compacted files will be deleted/cleared once all the existing readers on these compacted files are done.
    
    Specified by:
    
    getCompactedfiles in interface StoreFileManager
    
    Returns:
    
    the list of compacted files
  - getCompactedFilesCount
```
public int getCompactedFilesCount()
```
    Description copied from interface: StoreFileManager
    
    Returns the number of compacted files.
    
    Specified by:
    
    getCompactedFilesCount in interface StoreFileManager
    
    Returns:
    
    The number of files.
  - insertNewFiles
```
public void insertNewFiles(Collection<HStoreFile> sfs)
                    throws IOException
```
    Description copied from interface: StoreFileManager
    
    Adds new files, either for from MemStore flush or bulk insert, into the structure.
    
    Specified by:
    
    insertNewFiles in interface StoreFileManager
    
    Parameters:
    
    sfs - New store files.
    
    Throws:
    
    IOException
  - clearFiles
```
public org.apache.hbase.thirdparty.com.google.common.collect.ImmutableCollection<HStoreFile> clearFiles()
```
    Description copied from interface: StoreFileManager
    
    Clears all the files currently in use and returns them.
    
    Specified by:
    
    clearFiles in interface StoreFileManager
    
    Returns:
    
    The files previously in use.
  - clearCompactedFiles
```
public org.apache.hbase.thirdparty.com.google.common.collect.ImmutableCollection<HStoreFile> clearCompactedFiles()
```
    Description copied from interface: StoreFileManager
    
    Clears all the compacted files and returns them. This method is expected to be accessed single threaded.
    
    Specified by:
    
    clearCompactedFiles in interface StoreFileManager
    
    Returns:
    
    The files compacted previously.
  - getStorefileCount
```
public int getStorefileCount()
```
    Description copied from interface: StoreFileManager
    
    Returns the number of files currently in use.
    
    Specified by:
    
    getStorefileCount in interface StoreFileManager
    
    Returns:
    
    The number of files.
  - getCandidateFilesForRowKeyBefore
```
public Iterator<HStoreFile> getCandidateFilesForRowKeyBefore(KeyValue targetKey)
```
    See StoreFileManager.getCandidateFilesForRowKeyBefore(KeyValue) for details on this methods.
    
    Specified by:
    
    getCandidateFilesForRowKeyBefore in interface StoreFileManager
    
    Parameters:
    
    targetKey - The key that is the basis of the search.
    
    Returns:
    
    The files that may have the key less than or equal to targetKey, in reverse order of new-ness, and preference for target key.
  - updateCandidateFilesForRowKeyBefore
```
public Iterator<HStoreFile> updateCandidateFilesForRowKeyBefore(Iterator<HStoreFile> candidateFiles,
                                                                KeyValue targetKey,
                                                                Cell candidate)
```
    See StoreFileManager.getCandidateFilesForRowKeyBefore(KeyValue) and StoreFileManager.updateCandidateFilesForRowKeyBefore(Iterator, KeyValue, Cell) for details on this methods.
    
    Specified by:
    
    updateCandidateFilesForRowKeyBefore in interface StoreFileManager
    
    Parameters:
    
    candidateFiles - The candidate files not yet checked for better candidates - return value from StoreFileManager.getCandidateFilesForRowKeyBefore(KeyValue), with some files already removed.
    
    targetKey - The key to search for.
    
    candidate - The current best candidate found.
    
    Returns:
    
    The list to replace candidateFiles.
  - getSplitPoint
```
public Optional<byte[]> getSplitPoint()
                               throws IOException
```
    Override of getSplitPoint that determines the split point as the boundary between two stripes, unless it causes significant imbalance between split sides' sizes. In that case, the split boundary will be chosen from the middle of one of the stripes to minimize imbalance.
    
    Specified by:
    
    getSplitPoint in interface StoreFileManager
    
    Returns:
    
    The split point, or null if no split is possible.
    
    Throws:
    
    IOException
  - getSplitPointFromAllFiles
```
private Optional<byte[]> getSplitPointFromAllFiles()
                                            throws IOException
```
    Throws:
    
    IOException
  - getMidStripeSplitRatio
```
private double getMidStripeSplitRatio(long smallerSize,
                                      long largerSize,
                                      long lastLargerSize)
```
  - getFilesForScan
```
public Collection<HStoreFile> getFilesForScan(byte[] startRow,
                                              boolean includeStartRow,
                                              byte[] stopRow,
                                              boolean includeStopRow)
```
    Description copied from interface: StoreFileManager
    
    Gets the store files to scan for a Scan or Get request.
    
    Specified by:
    
    getFilesForScan in interface StoreFileManager
    
    Parameters:
    
    startRow - Start row of the request.
    
    stopRow - Stop row of the request.
    
    Returns:
    
    The list of files that are to be read for this request.
  - addCompactionResults
```
public void addCompactionResults(Collection<HStoreFile> compactedFiles,
                                 Collection<HStoreFile> results)
                          throws IOException
```
    Description copied from interface: StoreFileManager
    
    Adds only the new compaction results into the structure.
    
    Specified by:
    
    addCompactionResults in interface StoreFileManager
    
    Parameters:
    
    compactedFiles - The input files for the compaction.
    
    results - The resulting files for the compaction.
    
    Throws:
    
    IOException
  - markCompactedAway
```
private void markCompactedAway(Collection<HStoreFile> compactedFiles)
```
  - removeCompactedFiles
```
public void removeCompactedFiles(Collection<HStoreFile> compactedFiles)
                          throws IOException
```
    Description copied from interface: StoreFileManager
    
    Remove the compacted files
    
    Specified by:
    
    removeCompactedFiles in interface StoreFileManager
    
    Parameters:
    
    compactedFiles - the list of compacted files
    
    Throws:
    
    IOException
  - getStoreCompactionPriority
```
public int getStoreCompactionPriority()
```
    Specified by:
    
    getStoreCompactionPriority in interface StoreFileManager
    
    Returns:
    
    The store compaction priority.
  - getStripeFilesSize
```
private long getStripeFilesSize(int stripeIndex)
```
    Gets the total size of all files in the stripe.
    
    Parameters:
    
    stripeIndex - Stripe index.
    
    Returns:
    
    Size.
  - loadUnclassifiedStoreFiles
```
private void loadUnclassifiedStoreFiles(List<HStoreFile> storeFiles)
```
    Loads initial store files that were picked up from some physical location pertaining to this store (presumably). Unlike adding files after compaction, assumes empty initial sets, and is forgiving with regard to stripe constraints - at worst, many/all files will go to level 0.
    
    Parameters:
    
    storeFiles - Store files to add.
  - ensureEdgeStripeMetadata
```
private void ensureEdgeStripeMetadata(ArrayList<HStoreFile> stripe,
                                      boolean isFirst)
```
  - ensureLevel0Metadata
```
private void ensureLevel0Metadata(HStoreFile sf)
```
  - debugDumpState
```
private void debugDumpState(String string)
```
  - isOpen
```
private static final boolean isOpen(byte[] key)
```
    Checks whether the key indicates an open interval boundary (i.e. infinity).
  - isOpen
```
private static final boolean isOpen(Cell key)
```
  - isInvalid
```
private static final boolean isInvalid(byte[] key)
```
    Checks whether the key is invalid (e.g. from an L0 file, or non-stripe-compacted files).
  - rowEquals
```
private final boolean rowEquals(byte[] k1,
                                byte[] k2)
```
    Compare two keys for equality.
  - nonOpenRowCompare
```
private final int nonOpenRowCompare(byte[] k1,
                                    byte[] k2)
```
    Compare two keys. Keys must not be open (isOpen(row) == false).
  - nonOpenRowCompare
```
private final int nonOpenRowCompare(Cell k1,
                                    byte[] k2)
```
  - findStripeIndexByEndRow
```
private final int findStripeIndexByEndRow(byte[] endRow)
```
    Finds the stripe index by end row.
  - findStripeForRow
```
private final int findStripeForRow(byte[] row,
                                   boolean isStart)
```
    Finds the stripe index for the stripe containing a row provided externally for get/scan.
  - getStartRow
```
public final byte[] getStartRow(int stripeIndex)
```
    Description copied from interface: StripeCompactionPolicy.StripeInformationProvider
    
    Gets the start row for a given stripe.
    
    Specified by:
    
    getStartRow in interface StripeCompactionPolicy.StripeInformationProvider
    
    Parameters:
    
    stripeIndex - Stripe index.
    
    Returns:
    
    Start row. May be an open key.
  - getEndRow
```
public final byte[] getEndRow(int stripeIndex)
```
    Description copied from interface: StripeCompactionPolicy.StripeInformationProvider
    
    Gets the end row for a given stripe.
    
    Specified by:
    
    getEndRow in interface StripeCompactionPolicy.StripeInformationProvider
    
    Parameters:
    
    stripeIndex - Stripe index.
    
    Returns:
    
    End row. May be an open key.
  - startOf
```
private byte[] startOf(HStoreFile sf)
```
  - endOf
```
private byte[] endOf(HStoreFile sf)
```
  - insertFileIntoStripe
```
private static void insertFileIntoStripe(ArrayList<HStoreFile> stripe,
                                         HStoreFile sf)
```
    Inserts a file in the correct place (by seqnum) in a stripe copy.
    
    Parameters:
    
    stripe - Stripe copy to insert into.
    
    sf - File to insert.
  - getLevel0Files
```
public List<HStoreFile> getLevel0Files()
```
    Specified by:
    
    getLevel0Files in interface StripeCompactionPolicy.StripeInformationProvider
    
    Returns:
    
    Level 0 files.
  - getStripeBoundaries
```
public List<byte[]> getStripeBoundaries()
```
    Specified by:
    
    getStripeBoundaries in interface StripeCompactionPolicy.StripeInformationProvider
    
    Returns:
    
    All stripe boundaries; including the open ones on both ends.
  - getStripes
```
public ArrayList<org.apache.hbase.thirdparty.com.google.common.collect.ImmutableList<HStoreFile>> getStripes()
```
    Specified by:
    
    getStripes in interface StripeCompactionPolicy.StripeInformationProvider
    
    Returns:
    
    The stripes.
  - getStripeCount
```
public int getStripeCount()
```
    Specified by:
    
    getStripeCount in interface StripeCompactionPolicy.StripeInformationProvider
    
    Returns:
    
    Stripe count.
  - getUnneededFiles
```
public Collection<HStoreFile> getUnneededFiles(long maxTs,
                                               List<HStoreFile> filesCompacting)
```
    Specified by:
    
    getUnneededFiles in interface StoreFileManager
    
    Parameters:
    
    maxTs - Maximum expired timestamp.
    
    filesCompacting - Files that are currently compacting.
    
    Returns:
    
    The files which don't have any necessary data according to TTL and other criteria.
  - findExpiredFiles
```
private Collection<HStoreFile> findExpiredFiles(org.apache.hbase.thirdparty.com.google.common.collect.ImmutableList<HStoreFile> stripe,
                                                long maxTs,
                                                List<HStoreFile> filesCompacting,
                                                Collection<HStoreFile> expiredStoreFiles)
```
  - getCompactionPressure
```
public double getCompactionPressure()
```
    Specified by:
    
    getCompactionPressure in interface StoreFileManager
    
    Returns:
    
    the compaction pressure used for compaction throughput tuning.
    
    See Also:
    
    HStore.getCompactionPressure()
  - getStoreFileComparator
```
public Comparator<HStoreFile> getStoreFileComparator()
```
    Specified by:
    
    getStoreFileComparator in interface StoreFileManager
    
    Returns:
    
    the comparator used to sort storefiles. Usually, the HStoreFile.getMaxSequenceId() is the first priority.

Class StripeStoreFileManager

Nested Class Summary

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

LOG

STRIPE_START_KEY

STRIPE_END_KEY

MAP_COMPARATOR

OPEN_KEY

INVALID_KEY

state

fileStarts

fileEnds

INVALID_KEY_IN_MAP

cellComparator

config

blockingFileCount

Constructor Detail

StripeStoreFileManager

Method Detail

loadFiles

getStorefiles

getCompactedfiles

getCompactedFilesCount

insertNewFiles

clearFiles

clearCompactedFiles

getStorefileCount

getCandidateFilesForRowKeyBefore

updateCandidateFilesForRowKeyBefore

getSplitPoint

getSplitPointFromAllFiles

getMidStripeSplitRatio

getFilesForScan

addCompactionResults

markCompactedAway

removeCompactedFiles

getStoreCompactionPriority

getStripeFilesSize

loadUnclassifiedStoreFiles

ensureEdgeStripeMetadata

ensureLevel0Metadata

debugDumpState

isOpen

isOpen

isInvalid

rowEquals

nonOpenRowCompare

nonOpenRowCompare

findStripeIndexByEndRow

findStripeForRow

getStartRow

getEndRow

startOf

endOf

insertFileIntoStripe

getLevel0Files

getStripeBoundaries

getStripes

getStripeCount

getUnneededFiles

findExpiredFiles

getCompactionPressure

getStoreFileComparator