@InterfaceAudience.Private public class StripeStoreFileManager extends Object implements StoreFileManager, StripeCompactionPolicy.StripeInformationProvider
StoreFileManager
. Not thread safe - relies on external locking
(in HStore). Collections that this class returns are immutable or unique to the call, so they
should be safe. Stripe store splits the key space of the region into non-overlapping stripes, as
well as some recent files that have all the keys (level 0). Each stripe contains a set of files.
When L0 is compacted, it's split into the files corresponding to existing stripe boundaries, that
can thus be added to stripes. When scan or get happens, it only has to read the files from the
corresponding stripes. See StripeCompactionPolicy
on how the stripes are determined; this
class doesn't care. This class should work together with StripeCompactionPolicy
and
StripeCompactor
. With regard to how they
work, we make at least the following (reasonable) assumptions: - Compaction produces one file per
new stripe (if any); that is easy to change. - Compaction has one contiguous set of stripes both
in and out, except if L0 is involved.Modifier and Type | Class and Description |
---|---|
private class |
StripeStoreFileManager.CompactionOrFlushMergeCopy
Non-static helper class for merging compaction or flush results.
|
private static class |
StripeStoreFileManager.KeyBeforeConcatenatedLists
An extension of ConcatenatedLists that has several peculiar properties.
|
private static class |
StripeStoreFileManager.State
The state class.
|
Modifier and Type | Field and Description |
---|---|
private int |
blockingFileCount |
private CellComparator |
cellComparator |
private StripeStoreConfig |
config |
private HashMap<HStoreFile,byte[]> |
fileEnds |
private HashMap<HStoreFile,byte[]> |
fileStarts
Cached file metadata (or overrides as the case may be)
|
(package private) static byte[] |
INVALID_KEY |
private static byte[] |
INVALID_KEY_IN_MAP
Normally invalid key is null, but in the map null is the result for "no key"; so use the
following constant value in these maps instead.
|
private static org.slf4j.Logger |
LOG |
private static Bytes.RowEndKeyComparator |
MAP_COMPARATOR |
static byte[] |
OPEN_KEY
The key value used for range boundary, indicating that the boundary is open (i.e.
|
private StripeStoreFileManager.State |
state |
static byte[] |
STRIPE_END_KEY |
static byte[] |
STRIPE_START_KEY
The file metadata fields that contain the stripe information.
|
Constructor and Description |
---|
StripeStoreFileManager(CellComparator kvComparator,
org.apache.hadoop.conf.Configuration conf,
StripeStoreConfig config) |
Modifier and Type | Method and Description |
---|---|
void |
addCompactionResults(Collection<HStoreFile> compactedFiles,
Collection<HStoreFile> results)
Adds only the new compaction results into the structure.
|
org.apache.hbase.thirdparty.com.google.common.collect.ImmutableCollection<HStoreFile> |
clearCompactedFiles()
Clears all the compacted files and returns them.
|
org.apache.hbase.thirdparty.com.google.common.collect.ImmutableCollection<HStoreFile> |
clearFiles()
Clears all the files currently in use and returns them.
|
private void |
debugDumpState(String string) |
private byte[] |
endOf(HStoreFile sf) |
private void |
ensureEdgeStripeMetadata(ArrayList<HStoreFile> stripe,
boolean isFirst) |
private void |
ensureLevel0Metadata(HStoreFile sf) |
private Collection<HStoreFile> |
findExpiredFiles(org.apache.hbase.thirdparty.com.google.common.collect.ImmutableList<HStoreFile> stripe,
long maxTs,
List<HStoreFile> filesCompacting,
Collection<HStoreFile> expiredStoreFiles) |
private int |
findStripeForRow(byte[] row,
boolean isStart)
Finds the stripe index for the stripe containing a row provided externally for get/scan.
|
private int |
findStripeIndexByEndRow(byte[] endRow)
Finds the stripe index by end row.
|
Iterator<HStoreFile> |
getCandidateFilesForRowKeyBefore(KeyValue targetKey)
See
StoreFileManager.getCandidateFilesForRowKeyBefore(KeyValue) for details on this
methods. |
Collection<HStoreFile> |
getCompactedfiles()
List of compacted files inside this store that needs to be excluded in reads because further
new reads will be using only the newly created files out of compaction.
|
int |
getCompactedFilesCount()
Returns the number of compacted files.
|
double |
getCompactionPressure() |
byte[] |
getEndRow(int stripeIndex)
Gets the end row for a given stripe.
|
Collection<HStoreFile> |
getFilesForScan(byte[] startRow,
boolean includeStartRow,
byte[] stopRow,
boolean includeStopRow)
Gets the store files to scan for a Scan or Get request.
|
List<HStoreFile> |
getLevel0Files()
Returns Level 0 files.
|
private double |
getMidStripeSplitRatio(long smallerSize,
long largerSize,
long lastLargerSize) |
Optional<byte[]> |
getSplitPoint()
Override of getSplitPoint that determines the split point as the boundary between two stripes,
unless it causes significant imbalance between split sides' sizes.
|
private Optional<byte[]> |
getSplitPointFromAllFiles() |
byte[] |
getStartRow(int stripeIndex)
Gets the start row for a given stripe.
|
int |
getStoreCompactionPriority()
Returns The store compaction priority.
|
Comparator<HStoreFile> |
getStoreFileComparator() |
int |
getStorefileCount()
Returns the number of files currently in use.
|
Collection<HStoreFile> |
getStorefiles()
Gets the snapshot of the store files currently in use.
|
List<byte[]> |
getStripeBoundaries()
Returns All stripe boundaries; including the open ones on both ends.
|
int |
getStripeCount()
Returns Stripe count.
|
private long |
getStripeFilesSize(int stripeIndex)
Gets the total size of all files in the stripe.
|
ArrayList<org.apache.hbase.thirdparty.com.google.common.collect.ImmutableList<HStoreFile>> |
getStripes()
Returns The stripes.
|
Collection<HStoreFile> |
getUnneededFiles(long maxTs,
List<HStoreFile> filesCompacting) |
private static void |
insertFileIntoStripe(ArrayList<HStoreFile> stripe,
HStoreFile sf)
Inserts a file in the correct place (by seqnum) in a stripe copy.
|
void |
insertNewFiles(Collection<HStoreFile> sfs)
Adds new files, either for from MemStore flush or bulk insert, into the structure.
|
private static boolean |
isInvalid(byte[] key)
Checks whether the key is invalid (e.g.
|
private static boolean |
isOpen(byte[] key)
Checks whether the key indicates an open interval boundary (i.e.
|
private static boolean |
isOpen(Cell key) |
void |
loadFiles(List<HStoreFile> storeFiles)
Loads the initial store files into empty StoreFileManager.
|
private void |
loadUnclassifiedStoreFiles(List<HStoreFile> storeFiles)
Loads initial store files that were picked up from some physical location pertaining to this
store (presumably).
|
private void |
markCompactedAway(Collection<HStoreFile> compactedFiles) |
private int |
nonOpenRowCompare(byte[] k1,
byte[] k2)
Compare two keys.
|
private int |
nonOpenRowCompare(Cell k1,
byte[] k2) |
void |
removeCompactedFiles(Collection<HStoreFile> compactedFiles)
Remove the compacted files
|
private boolean |
rowEquals(byte[] k1,
byte[] k2)
Compare two keys for equality.
|
private byte[] |
startOf(HStoreFile sf) |
Iterator<HStoreFile> |
updateCandidateFilesForRowKeyBefore(Iterator<HStoreFile> candidateFiles,
KeyValue targetKey,
Cell candidate)
See
StoreFileManager.getCandidateFilesForRowKeyBefore(KeyValue) and
StoreFileManager.updateCandidateFilesForRowKeyBefore(Iterator, KeyValue, Cell) for
details on this methods. |
private static final org.slf4j.Logger LOG
public static final byte[] STRIPE_START_KEY
public static final byte[] STRIPE_END_KEY
private static final Bytes.RowEndKeyComparator MAP_COMPARATOR
public static final byte[] OPEN_KEY
static final byte[] INVALID_KEY
private StripeStoreFileManager.State state
private HashMap<HStoreFile,byte[]> fileStarts
private HashMap<HStoreFile,byte[]> fileEnds
private static final byte[] INVALID_KEY_IN_MAP
private final CellComparator cellComparator
private StripeStoreConfig config
private final int blockingFileCount
public StripeStoreFileManager(CellComparator kvComparator, org.apache.hadoop.conf.Configuration conf, StripeStoreConfig config)
public void loadFiles(List<HStoreFile> storeFiles)
StoreFileManager
loadFiles
in interface StoreFileManager
storeFiles
- The files to load.public Collection<HStoreFile> getStorefiles()
StoreFileManager
getStorefiles
in interface StripeCompactionPolicy.StripeInformationProvider
getStorefiles
in interface StoreFileManager
public Collection<HStoreFile> getCompactedfiles()
StoreFileManager
getCompactedfiles
in interface StoreFileManager
public int getCompactedFilesCount()
StoreFileManager
getCompactedFilesCount
in interface StoreFileManager
public void insertNewFiles(Collection<HStoreFile> sfs)
StoreFileManager
insertNewFiles
in interface StoreFileManager
sfs
- New store files.public org.apache.hbase.thirdparty.com.google.common.collect.ImmutableCollection<HStoreFile> clearFiles()
StoreFileManager
clearFiles
in interface StoreFileManager
public org.apache.hbase.thirdparty.com.google.common.collect.ImmutableCollection<HStoreFile> clearCompactedFiles()
StoreFileManager
clearCompactedFiles
in interface StoreFileManager
public int getStorefileCount()
StoreFileManager
getStorefileCount
in interface StoreFileManager
public Iterator<HStoreFile> getCandidateFilesForRowKeyBefore(KeyValue targetKey)
StoreFileManager.getCandidateFilesForRowKeyBefore(KeyValue)
for details on this
methods.getCandidateFilesForRowKeyBefore
in interface StoreFileManager
targetKey
- The key that is the basis of the search.public Iterator<HStoreFile> updateCandidateFilesForRowKeyBefore(Iterator<HStoreFile> candidateFiles, KeyValue targetKey, Cell candidate)
StoreFileManager.getCandidateFilesForRowKeyBefore(KeyValue)
and
StoreFileManager.updateCandidateFilesForRowKeyBefore(Iterator, KeyValue, Cell)
for
details on this methods.updateCandidateFilesForRowKeyBefore
in interface StoreFileManager
candidateFiles
- The candidate files not yet checked for better candidates - return value
from StoreFileManager.getCandidateFilesForRowKeyBefore(KeyValue)
, with some files
already removed.targetKey
- The key to search for.candidate
- The current best candidate found.public Optional<byte[]> getSplitPoint() throws IOException
getSplitPoint
in interface StoreFileManager
IOException
private Optional<byte[]> getSplitPointFromAllFiles() throws IOException
IOException
private double getMidStripeSplitRatio(long smallerSize, long largerSize, long lastLargerSize)
public Collection<HStoreFile> getFilesForScan(byte[] startRow, boolean includeStartRow, byte[] stopRow, boolean includeStopRow)
StoreFileManager
getFilesForScan
in interface StoreFileManager
startRow
- Start row of the request.stopRow
- Stop row of the request.public void addCompactionResults(Collection<HStoreFile> compactedFiles, Collection<HStoreFile> results)
StoreFileManager
addCompactionResults
in interface StoreFileManager
compactedFiles
- The input files for the compaction.results
- The resulting files for the compaction.private void markCompactedAway(Collection<HStoreFile> compactedFiles)
public void removeCompactedFiles(Collection<HStoreFile> compactedFiles)
StoreFileManager
removeCompactedFiles
in interface StoreFileManager
compactedFiles
- the list of compacted filespublic int getStoreCompactionPriority()
StoreFileManager
getStoreCompactionPriority
in interface StoreFileManager
private long getStripeFilesSize(int stripeIndex)
stripeIndex
- Stripe index.private void loadUnclassifiedStoreFiles(List<HStoreFile> storeFiles)
storeFiles
- Store files to add.private void ensureEdgeStripeMetadata(ArrayList<HStoreFile> stripe, boolean isFirst)
private void ensureLevel0Metadata(HStoreFile sf)
private void debugDumpState(String string)
private static final boolean isOpen(byte[] key)
private static final boolean isInvalid(byte[] key)
private final boolean rowEquals(byte[] k1, byte[] k2)
private final int nonOpenRowCompare(byte[] k1, byte[] k2)
private final int nonOpenRowCompare(Cell k1, byte[] k2)
private final int findStripeIndexByEndRow(byte[] endRow)
private final int findStripeForRow(byte[] row, boolean isStart)
public final byte[] getStartRow(int stripeIndex)
StripeCompactionPolicy.StripeInformationProvider
getStartRow
in interface StripeCompactionPolicy.StripeInformationProvider
stripeIndex
- Stripe index.public final byte[] getEndRow(int stripeIndex)
StripeCompactionPolicy.StripeInformationProvider
getEndRow
in interface StripeCompactionPolicy.StripeInformationProvider
stripeIndex
- Stripe index.private byte[] startOf(HStoreFile sf)
private byte[] endOf(HStoreFile sf)
private static void insertFileIntoStripe(ArrayList<HStoreFile> stripe, HStoreFile sf)
stripe
- Stripe copy to insert into.sf
- File to insert.public List<HStoreFile> getLevel0Files()
StripeCompactionPolicy.StripeInformationProvider
getLevel0Files
in interface StripeCompactionPolicy.StripeInformationProvider
public List<byte[]> getStripeBoundaries()
StripeCompactionPolicy.StripeInformationProvider
getStripeBoundaries
in interface StripeCompactionPolicy.StripeInformationProvider
public ArrayList<org.apache.hbase.thirdparty.com.google.common.collect.ImmutableList<HStoreFile>> getStripes()
StripeCompactionPolicy.StripeInformationProvider
getStripes
in interface StripeCompactionPolicy.StripeInformationProvider
public int getStripeCount()
StripeCompactionPolicy.StripeInformationProvider
getStripeCount
in interface StripeCompactionPolicy.StripeInformationProvider
public Collection<HStoreFile> getUnneededFiles(long maxTs, List<HStoreFile> filesCompacting)
getUnneededFiles
in interface StoreFileManager
maxTs
- Maximum expired timestamp.filesCompacting
- Files that are currently compacting.private Collection<HStoreFile> findExpiredFiles(org.apache.hbase.thirdparty.com.google.common.collect.ImmutableList<HStoreFile> stripe, long maxTs, List<HStoreFile> filesCompacting, Collection<HStoreFile> expiredStoreFiles)
public double getCompactionPressure()
getCompactionPressure
in interface StoreFileManager
HStore.getCompactionPressure()
public Comparator<HStoreFile> getStoreFileComparator()
getStoreFileComparator
in interface StoreFileManager
HStoreFile.getMaxSequenceId()
is the first priority.Copyright © 2007–2020 The Apache Software Foundation. All rights reserved.