Package org.apache.hadoop.hbase.snapshot
Class SnapshotManifest
java.lang.Object
org.apache.hadoop.hbase.snapshot.SnapshotManifest
Utility class to help read/write the Snapshot Manifest. The snapshot format is transparent for
the users of this class, once the snapshot is written, it will never be modified. On open() the
snapshot will be loaded to the current in-memory format.
-
Nested Class Summary
Modifier and TypeClassDescription(package private) static interface
-
Field Summary
Modifier and TypeFieldDescriptionprivate final org.apache.hadoop.conf.Configuration
static final String
private org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription
private TableDescriptor
private static final org.slf4j.Logger
private int
private final ForeignExceptionSnare
private List<org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotRegionManifest>
private final org.apache.hadoop.fs.FileSystem
static final String
private final MonitoredTask
private final org.apache.hadoop.fs.Path
private final org.apache.hadoop.fs.FileSystem
-
Constructor Summary
ModifierConstructorDescriptionprivate
SnapshotManifest
(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem rootFs, org.apache.hadoop.fs.Path workingDir, org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription desc, ForeignExceptionSnare monitor, MonitoredTask statusTask) -
Method Summary
Modifier and TypeMethodDescriptionvoid
addMobRegion
(RegionInfo regionInfo) protected void
addMobRegion
(RegionInfo regionInfo, SnapshotManifest.RegionVisitor visitor) private void
addReferenceFiles
(SnapshotManifest.RegionVisitor visitor, Object regionData, Object familyData, Collection<StoreFileInfo> storeFiles, boolean isMob) void
addRegion
(org.apache.hadoop.fs.Path tableDir, RegionInfo regionInfo) Creates a 'manifest' for the specified region, by reading directly from the disk.protected void
addRegion
(org.apache.hadoop.fs.Path tableDir, RegionInfo regionInfo, SnapshotManifest.RegionVisitor visitor) void
Creates a 'manifest' for the specified region, by reading directly from the HRegion object.protected void
addRegion
(HRegion region, SnapshotManifest.RegionVisitor visitor) void
Add the table descriptor to the snapshot manifestvoid
private void
static SnapshotManifest
create
(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path workingDir, org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription desc, ForeignExceptionSnare monitor) Return a SnapshotManifest instance, used for writing a snapshot.static SnapshotManifest
create
(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path workingDir, org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription desc, ForeignExceptionSnare monitor, MonitoredTask statusTask) private ThreadPoolExecutor
createExecutor
(String name) static ThreadPoolExecutor
createExecutor
(org.apache.hadoop.conf.Configuration conf, String name) private SnapshotManifest.RegionVisitor
createRegionVisitor
(org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription desc) List<org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotRegionManifest>
Get all the Region Manifest from the snapshotGet all the Region Manifest from the snapshot.(package private) static String
getRegionNameFromManifest
(org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotRegionManifest manifest) Extract the region encoded name from the region manifestorg.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription
Get the SnapshotDescriptionorg.apache.hadoop.fs.Path
Get the current snapshot working dirprivate static int
getSnapshotFormat
(org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription desc) private List<StoreFileInfo>
getStoreFiles
(org.apache.hadoop.fs.Path storeDir) Get the table descriptor from the Snapshotprivate void
load()
Load the information in the SnapshotManifest.static SnapshotManifest
open
(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path workingDir, org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription desc) Return a SnapshotManifest instance with the information already loaded in-memory.private org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDataManifest
private void
setStatusMsg
(String msg) private void
writeDataManifest
(org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDataManifest manifest)
-
Field Details
-
LOG
-
SNAPSHOT_MANIFEST_SIZE_LIMIT_CONF_KEY
- See Also:
-
DATA_MANIFEST_NAME
- See Also:
-
regionManifests
private List<org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotRegionManifest> regionManifests -
desc
-
htd
-
monitor
-
conf
-
workingDir
-
rootFs
-
workingDirFs
-
manifestSizeLimit
-
statusTask
-
-
Constructor Details
-
SnapshotManifest
private SnapshotManifest(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem rootFs, org.apache.hadoop.fs.Path workingDir, org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription desc, ForeignExceptionSnare monitor, MonitoredTask statusTask) throws IOException - Parameters:
conf
- configuration file for HBase setuprootFs
- root filesystem containing HFilesworkingDir
- file path of where the manifest should be locateddesc
- description of snapshot being takenmonitor
- monitor of foreign exceptions- Throws:
IOException
- if the working directory file system cannot be determined from the config file
-
-
Method Details
-
create
public static SnapshotManifest create(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path workingDir, org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription desc, ForeignExceptionSnare monitor) throws IOException Return a SnapshotManifest instance, used for writing a snapshot. There are two usage pattern: - The Master will create a manifest, add the descriptor, offline regions and consolidate the snapshot by writing all the pending stuff on-disk. manifest = SnapshotManifest.create(...) manifest.addRegion(tableDir, hri) manifest.consolidate() - The RegionServer will create a single region manifest manifest = SnapshotManifest.create(...) manifest.addRegion(region)- Throws:
IOException
-
create
public static SnapshotManifest create(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path workingDir, org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription desc, ForeignExceptionSnare monitor, MonitoredTask statusTask) throws IOException - Throws:
IOException
-
open
public static SnapshotManifest open(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path workingDir, org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription desc) throws IOException Return a SnapshotManifest instance with the information already loaded in-memory. SnapshotManifest manifest = SnapshotManifest.open(...) TableDescriptor htd = manifest.getDescriptor() for (SnapshotRegionManifest regionManifest: manifest.getRegionManifests()) hri = regionManifest.getRegionInfo() for (regionManifest.getFamilyFiles()) ...- Throws:
IOException
-
addTableDescriptor
Add the table descriptor to the snapshot manifest- Throws:
IOException
-
createRegionVisitor
private SnapshotManifest.RegionVisitor createRegionVisitor(org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription desc) throws IOException - Throws:
IOException
-
addMobRegion
- Throws:
IOException
-
addMobRegion
protected void addMobRegion(RegionInfo regionInfo, SnapshotManifest.RegionVisitor visitor) throws IOException - Throws:
IOException
-
addRegion
Creates a 'manifest' for the specified region, by reading directly from the HRegion object. This is used by the "online snapshot" when the table is enabled.- Throws:
IOException
-
addRegion
- Throws:
IOException
-
addRegion
Creates a 'manifest' for the specified region, by reading directly from the disk. This is used by the "offline snapshot" when the table is disabled.- Throws:
IOException
-
addRegion
protected void addRegion(org.apache.hadoop.fs.Path tableDir, RegionInfo regionInfo, SnapshotManifest.RegionVisitor visitor) throws IOException - Throws:
IOException
-
getStoreFiles
- Throws:
IOException
-
addReferenceFiles
private void addReferenceFiles(SnapshotManifest.RegionVisitor visitor, Object regionData, Object familyData, Collection<StoreFileInfo> storeFiles, boolean isMob) throws IOException - Throws:
IOException
-
load
Load the information in the SnapshotManifest. Called by SnapshotManifest.open() If the format is v2 and there is no data-manifest, means that we are loading an in-progress snapshot. Since we support rolling-upgrades, we loook for v1 and v2 regions format.- Throws:
IOException
-
getSnapshotDir
Get the current snapshot working dir -
getSnapshotDescription
public org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription getSnapshotDescription()Get the SnapshotDescription -
getTableDescriptor
Get the table descriptor from the Snapshot -
getRegionManifests
public List<org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotRegionManifest> getRegionManifests()Get all the Region Manifest from the snapshot -
setStatusMsg
-
getRegionManifestsMap
public Map<String,org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotRegionManifest> getRegionManifestsMap()Get all the Region Manifest from the snapshot. This is an helper to get a map with the region encoded name -
consolidate
- Throws:
IOException
-
convertToV2SingleManifest
- Throws:
IOException
-
writeDataManifest
private void writeDataManifest(org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDataManifest manifest) throws IOException - Throws:
IOException
-
readDataManifest
private org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDataManifest readDataManifest() throws IOException- Throws:
IOException
-
createExecutor
-
createExecutor
public static ThreadPoolExecutor createExecutor(org.apache.hadoop.conf.Configuration conf, String name) -
getRegionNameFromManifest
static String getRegionNameFromManifest(org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotRegionManifest manifest) Extract the region encoded name from the region manifest -
getSnapshotFormat
private static int getSnapshotFormat(org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription desc)
-