Interface StoreFileTracker
- All Known Implementing Classes:
DefaultStoreFileTracker
,FileBasedStoreFileTracker
,MigrationStoreFileTracker
,StoreFileTrackerBase
An interface to define how we track the store files for a give store.
In the old time, we will write store to a tmp directory first, and then rename it to the actual
data file. And once a store file is under data directory, we will consider it as 'committed'. And
we need to do listing when loading store files.
When cloud age is coming, now we want to store the store files on object storage, where rename
and list are not as cheap as on HDFS, especially rename. Although introducing a metadata
management layer for object storage could solve the problem, but we still want HBase to run on
pure object storage, so here we introduce this interface to abstract how we track the store
files. For the old implementation, we just persist nothing here, and do listing to load store
files. When running on object storage, we could persist the store file list in a system region,
or in a file on the object storage, to make it possible to write directly into the data directory
to avoid renaming, and also avoid listing when loading store files.
The implementation requires to be thread safe as flush and compaction may occur as the same time,
and we could also do multiple compactions at the same time. As the implementation may choose to
persist the store file list to external storage, which could be slow, it is the duty for the
callers to not call it inside a lock which may block normal read/write requests.
-
Method Summary
Modifier and TypeMethodDescriptionvoid
add
(Collection<StoreFileInfo> newFiles) Add new store files.createReference
(Reference reference, org.apache.hadoop.fs.Path path) Create a writer for writing new store files.getStoreFileInfo
(org.apache.hadoop.fs.FileStatus fileStatus, org.apache.hadoop.fs.Path initialPath, boolean primaryReplica) getStoreFileInfo
(org.apache.hadoop.fs.Path initialPath, boolean primaryReplica) boolean
Returns true if the specified family has reference filesload()
Load the store files list when opening a region.readReference
(org.apache.hadoop.fs.Path path) Reads the reference file from the given path.void
replace
(Collection<StoreFileInfo> compactedFiles, Collection<StoreFileInfo> newFiles) Add new store files and remove compacted store files after compaction.boolean
Whether the implementation of this tracker requires you to write to temp directory first, i.e, does not allow broken store files under the actual data directory.void
set
(List<StoreFileInfo> files) Set the store files.Adds StoreFileTracker implementations specific configurations into the table descriptor.
-
Method Details
-
load
Load the store files list when opening a region.- Throws:
IOException
-
add
Add new store files. Used for flush and bulk load.- Throws:
IOException
-
replace
void replace(Collection<StoreFileInfo> compactedFiles, Collection<StoreFileInfo> newFiles) throws IOException Add new store files and remove compacted store files after compaction.- Throws:
IOException
-
set
Set the store files.- Throws:
IOException
-
createWriter
Create a writer for writing new store files.- Returns:
- Writer for a new StoreFile
- Throws:
IOException
-
updateWithTrackerConfigs
Adds StoreFileTracker implementations specific configurations into the table descriptor. This is used to avoid accidentally data loss when changing the cluster level store file tracker implementation, and also possible misconfiguration between master and region servers. See HBASE-26246 for more details.- Parameters:
builder
- The table descriptor builder for the given table.
-
requireWritingToTmpDirFirst
boolean requireWritingToTmpDirFirst()Whether the implementation of this tracker requires you to write to temp directory first, i.e, does not allow broken store files under the actual data directory. -
createReference
- Throws:
IOException
-
readReference
Reads the reference file from the given path.- Parameters:
path
- thePath
to the reference file in the file system.- Returns:
- a
Reference
that points at top/bottom half of a an hfile - Throws:
IOException
-
hasReferences
Returns true if the specified family has reference files- Returns:
- true if family contains reference files
- Throws:
IOException
-
getStoreFileInfo
StoreFileInfo getStoreFileInfo(org.apache.hadoop.fs.FileStatus fileStatus, org.apache.hadoop.fs.Path initialPath, boolean primaryReplica) throws IOException - Throws:
IOException
-
getStoreFileInfo
StoreFileInfo getStoreFileInfo(org.apache.hadoop.fs.Path initialPath, boolean primaryReplica) throws IOException - Throws:
IOException
-