StripeStoreFileManager (Apache HBase 1.2.12 API)

java.lang.Object
- org.apache.hadoop.hbase.regionserver.StripeStoreFileManager

All Implemented Interfaces:

StripeCompactionPolicy.StripeInformationProvider, StoreFileManager
```
@InterfaceAudience.Private
public class StripeStoreFileManager
extends Object
implements StoreFileManager, StripeCompactionPolicy.StripeInformationProvider
```
Stripe implementation of StoreFileManager. Not thread safe - relies on external locking (in HStore). Collections that this class returns are immutable or unique to the call, so they should be safe. Stripe store splits the key space of the region into non-overlapping stripes, as well as some recent files that have all the keys (level 0). Each stripe contains a set of files. When L0 is compacted, it's split into the files corresponding to existing stripe boundaries, that can thus be added to stripes. When scan or get happens, it only has to read the files from the corresponding stripes. See StripeCompationPolicy on how the stripes are determined; this class doesn't care. This class should work together with StripeCompactionPolicy and StripeCompactor. With regard to how they work, we make at least the following (reasonable) assumptions: - Compaction produces one file per new stripe (if any); that is easy to change. - Compaction has one contiguous set of stripes both in and out, except if L0 is involved.

Nested Class Summary

Nested Classes
Modifier and Type	Class and Description
`private class`	`StripeStoreFileManager.CompactionOrFlushMergeCopy` Non-static helper class for merging compaction or flush results.
`private static class`	`StripeStoreFileManager.KeyBeforeConcatenatedLists` An extension of ConcatenatedLists that has several peculiar properties.
`private static class`	`StripeStoreFileManager.State` The state class.

Field Summary

Fields
Modifier and Type	Field and Description
`private int`	`blockingFileCount`
`private StripeStoreConfig`	`config`
`private HashMap<StoreFile,byte[]>`	`fileEnds`
`private HashMap<StoreFile,byte[]>`	`fileStarts` Cached file metadata (or overrides as the case may be)
`(package private) static byte[]`	`INVALID_KEY`
`private static byte[]`	`INVALID_KEY_IN_MAP` Normally invalid key is null, but in the map null is the result for "no key"; so use the following constant value in these maps instead.
`private KeyValue.KVComparator`	`kvComparator`
`private static org.apache.commons.logging.Log`	`LOG`
`private static Bytes.RowEndKeyComparator`	`MAP_COMPARATOR`
`static byte[]`	`OPEN_KEY` The key value used for range boundary, indicating that the boundary is open (i.e.
`private StripeStoreFileManager.State`	`state`
`static byte[]`	`STRIPE_END_KEY`
`static byte[]`	`STRIPE_START_KEY` The file metadata fields that contain the stripe information.

Constructor Summary

Constructors
Constructor and Description
`StripeStoreFileManager(KeyValue.KVComparator kvComparator, org.apache.hadoop.conf.Configuration conf, StripeStoreConfig config)`

Method Summary

Methods
Modifier and Type	Method and Description
`void`	`addCompactionResults(Collection<StoreFile> compactedFiles, Collection<StoreFile> results)` Adds compaction results into the structure.
`com.google.common.collect.ImmutableCollection<StoreFile>`	`clearFiles()` Clears all the files currently in use and returns them.
`private void`	`debugDumpState(String string)`
`private byte[]`	`endOf(StoreFile sf)`
`private void`	`ensureEdgeStripeMetadata(ArrayList<StoreFile> stripe, boolean isFirst)`
`private void`	`ensureLevel0Metadata(StoreFile sf)`
`private Collection<StoreFile>`	`findExpiredFiles(com.google.common.collect.ImmutableList<StoreFile> stripe, long maxTs, List<StoreFile> filesCompacting, Collection<StoreFile> expiredStoreFiles)`
`private int`	`findStripeForRow(byte[] row, boolean isStart)` Finds the stripe index for the stripe containing a row provided externally for get/scan.
`private int`	`findStripeIndexByEndRow(byte[] endRow)` Finds the stripe index by end row.
`Iterator<StoreFile>`	`getCandidateFilesForRowKeyBefore(KeyValue targetKey)` See `StoreFileManager.getCandidateFilesForRowKeyBefore(KeyValue)` for details on this methods.
`double`	`getCompactionPressure()`
`byte[]`	`getEndRow(int stripeIndex)` Gets the end row for a given stripe.
`Collection<StoreFile>`	`getFilesForScanOrGet(boolean isGet, byte[] startRow, byte[] stopRow)` Gets the store files to scan for a Scan or Get request.
`List<StoreFile>`	`getLevel0Files()`
`private double`	`getMidStripeSplitRatio(long smallerSize, long largerSize, long lastLargerSize)`
`byte[]`	`getSplitPoint()` Gets the split point for the split of this set of store files (approx.
`private byte[]`	`getSplitPointFromAllFiles()`
`byte[]`	`getStartRow(int stripeIndex)` Gets the start row for a given stripe.
`int`	`getStoreCompactionPriority()`
`int`	`getStorefileCount()` Returns the number of files currently in use.
`Collection<StoreFile>`	`getStorefiles()` Gets the snapshot of the store files currently in use.
`List<byte[]>`	`getStripeBoundaries()`
`int`	`getStripeCount()`
`private long`	`getStripeFilesSize(int stripeIndex)` Gets the total size of all files in the stripe.
`ArrayList<com.google.common.collect.ImmutableList<StoreFile>>`	`getStripes()`
`Collection<StoreFile>`	`getUnneededFiles(long maxTs, List<StoreFile> filesCompacting)`
`private static void`	`insertFileIntoStripe(ArrayList<StoreFile> stripe, StoreFile sf)` Inserts a file in the correct place (by seqnum) in a stripe copy.
`void`	`insertNewFiles(Collection<StoreFile> sfs)` Adds new files, either for from MemStore flush or bulk insert, into the structure.
`private static boolean`	`isInvalid(byte[] key)` Checks whether the key is invalid (e.g.
`private static boolean`	`isOpen(byte[] key)` Checks whether the key indicates an open interval boundary (i.e.
`void`	`loadFiles(List<StoreFile> storeFiles)` Loads the initial store files into empty StoreFileManager.
`private void`	`loadUnclassifiedStoreFiles(List<StoreFile> storeFiles)` Loads initial store files that were picked up from some physical location pertaining to this store (presumably).
`private int`	`nonOpenRowCompare(byte[] k1, byte[] k2)` Compare two keys.
`private boolean`	`rowEquals(byte[] k1, byte[] k2)` Compare two keys for equality.
`private byte[]`	`startOf(StoreFile sf)`
`Iterator<StoreFile>`	`updateCandidateFilesForRowKeyBefore(Iterator<StoreFile> candidateFiles, KeyValue targetKey, Cell candidate)` See `StoreFileManager.getCandidateFilesForRowKeyBefore(KeyValue)` and `StoreFileManager.updateCandidateFilesForRowKeyBefore(Iterator, KeyValue, Cell)` for details on this methods.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - LOG
```
private static final org.apache.commons.logging.Log LOG
```
  - STRIPE_START_KEY
```
public static final byte[] STRIPE_START_KEY
```
    The file metadata fields that contain the stripe information.
  - STRIPE_END_KEY
```
public static final byte[] STRIPE_END_KEY
```
  - MAP_COMPARATOR
```
private static final Bytes.RowEndKeyComparator MAP_COMPARATOR
```
  - OPEN_KEY
```
public static final byte[] OPEN_KEY
```
    The key value used for range boundary, indicating that the boundary is open (i.e. +-inf).
  - INVALID_KEY
```
static final byte[] INVALID_KEY
```
  - state
```
private StripeStoreFileManager.State state
```
  - fileStarts
```
private HashMap<StoreFile,byte[]> fileStarts
```
    Cached file metadata (or overrides as the case may be)
  - fileEnds
```
private HashMap<StoreFile,byte[]> fileEnds
```
  - INVALID_KEY_IN_MAP
```
private static final byte[] INVALID_KEY_IN_MAP
```
    Normally invalid key is null, but in the map null is the result for "no key"; so use the following constant value in these maps instead. Note that this is a constant and we use it to compare by reference when we read from the map.
  - kvComparator
```
private final KeyValue.KVComparator kvComparator
```
  - config
```
private StripeStoreConfig config
```
  - blockingFileCount
```
private final int blockingFileCount
```
- Constructor Detail
  - StripeStoreFileManager
```
public StripeStoreFileManager(KeyValue.KVComparator kvComparator,
                      org.apache.hadoop.conf.Configuration conf,
                      StripeStoreConfig config)
```
- Method Detail
  - loadFiles
```
public void loadFiles(List<StoreFile> storeFiles)
```
    Description copied from interface: StoreFileManager
    
    Loads the initial store files into empty StoreFileManager.
    
    Specified by:
    
    loadFiles in interface StoreFileManager
    
    Parameters:
    storeFiles - The files to load.
  - getStorefiles
```
public Collection<StoreFile> getStorefiles()
```
    Description copied from interface: StoreFileManager
    
    Gets the snapshot of the store files currently in use. Can be used for things like metrics and checks; should not assume anything about relations between store files in the list.
    
    Specified by:
    
    getStorefiles in interface StripeCompactionPolicy.StripeInformationProvider
    
    Specified by:
    
    getStorefiles in interface StoreFileManager
    
    Returns:
    The list of StoreFiles.
  - insertNewFiles
```
public void insertNewFiles(Collection<StoreFile> sfs)
                    throws IOException
```
    Description copied from interface: StoreFileManager
    
    Adds new files, either for from MemStore flush or bulk insert, into the structure.
    
    Specified by:
    
    insertNewFiles in interface StoreFileManager
    
    Parameters:
    sfs - New store files.
    
    Throws:
    
    IOException
  - clearFiles
```
public com.google.common.collect.ImmutableCollection<StoreFile> clearFiles()
```
    Description copied from interface: StoreFileManager
    
    Clears all the files currently in use and returns them.
    
    Specified by:
    
    clearFiles in interface StoreFileManager
    
    Returns:
    The files previously in use.
  - getStorefileCount
```
public int getStorefileCount()
```
    Description copied from interface: StoreFileManager
    
    Returns the number of files currently in use.
    
    Specified by:
    
    getStorefileCount in interface StoreFileManager
    
    Returns:
    The number of files.
  - getCandidateFilesForRowKeyBefore
```
public Iterator<StoreFile> getCandidateFilesForRowKeyBefore(KeyValue targetKey)
```
    See StoreFileManager.getCandidateFilesForRowKeyBefore(KeyValue) for details on this methods.
    
    Specified by:
    
    getCandidateFilesForRowKeyBefore in interface StoreFileManager
    
    Parameters:
    targetKey - The key that is the basis of the search.
    
    Returns:
    The files that may have the key less than or equal to targetKey, in reverse order of new-ness, and preference for target key.
  - updateCandidateFilesForRowKeyBefore
```
public Iterator<StoreFile> updateCandidateFilesForRowKeyBefore(Iterator<StoreFile> candidateFiles,
                                                      KeyValue targetKey,
                                                      Cell candidate)
```
    See StoreFileManager.getCandidateFilesForRowKeyBefore(KeyValue) and StoreFileManager.updateCandidateFilesForRowKeyBefore(Iterator, KeyValue, Cell) for details on this methods.
    
    Specified by:
    
    updateCandidateFilesForRowKeyBefore in interface StoreFileManager
    
    Parameters:
    candidateFiles - The candidate files not yet checked for better candidates - return value from StoreFileManager.getCandidateFilesForRowKeyBefore(KeyValue), with some files already removed.
    targetKey - The key to search for.
    candidate - The current best candidate found.
    
    Returns:
    The list to replace candidateFiles.
  - getSplitPoint
```
public byte[] getSplitPoint()
                     throws IOException
```
    Description copied from interface: StoreFileManager
    
    Gets the split point for the split of this set of store files (approx. middle).
    
    Specified by:
    
    getSplitPoint in interface StoreFileManager
    
    Returns:
    The mid-point, or null if no split is possible.
    
    Throws:
    
    IOException
  - getSplitPointFromAllFiles
```
private byte[] getSplitPointFromAllFiles()
                                  throws IOException
```
    Throws:
    
    IOException
  - getMidStripeSplitRatio
```
private double getMidStripeSplitRatio(long smallerSize,
                            long largerSize,
                            long lastLargerSize)
```
  - getFilesForScanOrGet
```
public Collection<StoreFile> getFilesForScanOrGet(boolean isGet,
                                         byte[] startRow,
                                         byte[] stopRow)
```
    Description copied from interface: StoreFileManager
    
    Gets the store files to scan for a Scan or Get request.
    
    Specified by:
    
    getFilesForScanOrGet in interface StoreFileManager
    
    Parameters:
    isGet - Whether it's a get.
    startRow - Start row of the request.
    stopRow - Stop row of the request.
    
    Returns:
    The list of files that are to be read for this request.
  - addCompactionResults
```
public void addCompactionResults(Collection<StoreFile> compactedFiles,
                        Collection<StoreFile> results)
                          throws IOException
```
    Description copied from interface: StoreFileManager
    
    Adds compaction results into the structure.
    
    Specified by:
    
    addCompactionResults in interface StoreFileManager
    
    Parameters:
    compactedFiles - The input files for the compaction.
    results - The resulting files for the compaction.
    
    Throws:
    
    IOException
  - getStoreCompactionPriority
```
public int getStoreCompactionPriority()
```
    Specified by:
    
    getStoreCompactionPriority in interface StoreFileManager
    
    Returns:
    The store compaction priority.
  - getStripeFilesSize
```
private long getStripeFilesSize(int stripeIndex)
```
    Gets the total size of all files in the stripe.
    
    Parameters:
    stripeIndex - Stripe index.
    
    Returns:
    Size.
  - loadUnclassifiedStoreFiles
```
private void loadUnclassifiedStoreFiles(List<StoreFile> storeFiles)
```
    Loads initial store files that were picked up from some physical location pertaining to this store (presumably). Unlike adding files after compaction, assumes empty initial sets, and is forgiving with regard to stripe constraints - at worst, many/all files will go to level 0.
    
    Parameters:
    storeFiles - Store files to add.
  - ensureEdgeStripeMetadata
```
private void ensureEdgeStripeMetadata(ArrayList<StoreFile> stripe,
                            boolean isFirst)
```
  - ensureLevel0Metadata
```
private void ensureLevel0Metadata(StoreFile sf)
```
  - debugDumpState
```
private void debugDumpState(String string)
```
  - isOpen
```
private static final boolean isOpen(byte[] key)
```
    Checks whether the key indicates an open interval boundary (i.e. infinity).
  - isInvalid
```
private static final boolean isInvalid(byte[] key)
```
    Checks whether the key is invalid (e.g. from an L0 file, or non-stripe-compacted files).
  - rowEquals
```
private final boolean rowEquals(byte[] k1,
                byte[] k2)
```
    Compare two keys for equality.
  - nonOpenRowCompare
```
private final int nonOpenRowCompare(byte[] k1,
                    byte[] k2)
```
    Compare two keys. Keys must not be open (isOpen(row) == false).
  - findStripeIndexByEndRow
```
private final int findStripeIndexByEndRow(byte[] endRow)
```
    Finds the stripe index by end row.
  - findStripeForRow
```
private final int findStripeForRow(byte[] row,
                   boolean isStart)
```
    Finds the stripe index for the stripe containing a row provided externally for get/scan.
  - getStartRow
```
public final byte[] getStartRow(int stripeIndex)
```
    Description copied from interface: StripeCompactionPolicy.StripeInformationProvider
    
    Gets the start row for a given stripe.
    
    Specified by:
    
    getStartRow in interface StripeCompactionPolicy.StripeInformationProvider
    
    Parameters:
    stripeIndex - Stripe index.
    
    Returns:
    Start row. May be an open key.
  - getEndRow
```
public final byte[] getEndRow(int stripeIndex)
```
    Description copied from interface: StripeCompactionPolicy.StripeInformationProvider
    
    Gets the end row for a given stripe.
    
    Specified by:
    
    getEndRow in interface StripeCompactionPolicy.StripeInformationProvider
    
    Parameters:
    stripeIndex - Stripe index.
    
    Returns:
    End row. May be an open key.
  - startOf
```
private byte[] startOf(StoreFile sf)
```
  - endOf
```
private byte[] endOf(StoreFile sf)
```
  - insertFileIntoStripe
```
private static void insertFileIntoStripe(ArrayList<StoreFile> stripe,
                        StoreFile sf)
```
    Inserts a file in the correct place (by seqnum) in a stripe copy.
    
    Parameters:
    stripe - Stripe copy to insert into.
    sf - File to insert.
  - getLevel0Files
```
public List<StoreFile> getLevel0Files()
```
    Specified by:
    
    getLevel0Files in interface StripeCompactionPolicy.StripeInformationProvider
    
    Returns:
    Level 0 files.
  - getStripeBoundaries
```
public List<byte[]> getStripeBoundaries()
```
    Specified by:
    
    getStripeBoundaries in interface StripeCompactionPolicy.StripeInformationProvider
    
    Returns:
    All stripe boundaries; including the open ones on both ends.
  - getStripes
```
public ArrayList<com.google.common.collect.ImmutableList<StoreFile>> getStripes()
```
    Specified by:
    
    getStripes in interface StripeCompactionPolicy.StripeInformationProvider
    
    Returns:
    The stripes.
  - getStripeCount
```
public int getStripeCount()
```
    Specified by:
    
    getStripeCount in interface StripeCompactionPolicy.StripeInformationProvider
    
    Returns:
    Stripe count.
  - getUnneededFiles
```
public Collection<StoreFile> getUnneededFiles(long maxTs,
                                     List<StoreFile> filesCompacting)
```
    Specified by:
    
    getUnneededFiles in interface StoreFileManager
    
    Parameters:
    maxTs - Maximum expired timestamp.
    filesCompacting - Files that are currently compacting.
    
    Returns:
    The files which don't have any necessary data according to TTL and other criteria.
  - findExpiredFiles
```
private Collection<StoreFile> findExpiredFiles(com.google.common.collect.ImmutableList<StoreFile> stripe,
                                     long maxTs,
                                     List<StoreFile> filesCompacting,
                                     Collection<StoreFile> expiredStoreFiles)
```
  - getCompactionPressure
```
public double getCompactionPressure()
```
    Specified by:
    
    getCompactionPressure in interface StoreFileManager
    
    Returns:
    the compaction pressure used for compaction throughput tuning.
    See Also:
    Store.getCompactionPressure()

Class StripeStoreFileManager

Nested Class Summary

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

LOG

STRIPE_START_KEY

STRIPE_END_KEY

MAP_COMPARATOR

OPEN_KEY

INVALID_KEY

state

fileStarts

fileEnds

INVALID_KEY_IN_MAP

kvComparator

config

blockingFileCount

Constructor Detail

StripeStoreFileManager

Method Detail

loadFiles

getStorefiles

insertNewFiles

clearFiles

getStorefileCount

getCandidateFilesForRowKeyBefore

updateCandidateFilesForRowKeyBefore

getSplitPoint

getSplitPointFromAllFiles

getMidStripeSplitRatio

getFilesForScanOrGet

addCompactionResults

getStoreCompactionPriority

getStripeFilesSize

loadUnclassifiedStoreFiles

ensureEdgeStripeMetadata

ensureLevel0Metadata

debugDumpState

isOpen

isInvalid

rowEquals

nonOpenRowCompare

findStripeIndexByEndRow

findStripeForRow

getStartRow

getEndRow

startOf

endOf

insertFileIntoStripe

getLevel0Files

getStripeBoundaries

getStripes

getStripeCount

getUnneededFiles

findExpiredFiles

getCompactionPressure