org.apache.hadoop.hbase.snapshot

Class RestoreSnapshotHelper

java.lang.Object

org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper

@InterfaceAudience.Private
public class RestoreSnapshotHelper
extends Object

Helper to Restore/Clone a Snapshot

The helper assumes that a table is already created, and by calling restore() the content present in the snapshot will be restored as the new content of the table.

Clone from Snapshot: If the target table is empty, the restore operation is just a "clone operation", where the only operations are:

• for each region in the snapshot create a new region (note that the region will have a different name, since the encoding contains the table name)
• for each file in the region create a new HFileLink to point to the original file.
• restore the logs, if any

Restore from Snapshot:

• for each region in the table verify which are available in the snapshot and which are not
  • if the region is not present in the snapshot, remove it.
  • if the region is present in the snapshot
    • for each file in the table region verify which are available in the snapshot
      • if the hfile is not present in the snapshot, remove it
      • if the hfile is present, keep it (nothing to do)
    • for each file in the snapshot region but not in the table
      • create a new HFileLink that point to the original file
• for each region in the snapshot not present in the current table state
  • create a new region and for each file in the region create a new HFileLink (This is the same as the clone operation)
• restore the logs, if any

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	RestoreSnapshotHelper.RestoreMetaChanges Describe the set of operations needed to update hbase:meta after restore.

Field Summary

Fields

Modifier and Type	Field and Description
private org.apache.hadoop.conf.Configuration	conf
private boolean	createBackRefs
private org.apache.hadoop.fs.FileSystem	fs
private static org.slf4j.Logger	LOG
private ForeignExceptionDispatcher	monitor
private Map<String,Pair<String,String>>	parentsMap
private Map<byte[],byte[]>	regionsMap
private org.apache.hadoop.fs.Path	rootDir
private org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription	snapshotDesc
private SnapshotManifest	snapshotManifest
private TableName	snapshotTable
private MonitoredTask	status
private TableDescriptor	tableDesc
private org.apache.hadoop.fs.Path	tableDir

Constructor Summary

Constructors

Constructor and Description
RestoreSnapshotHelper(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, SnapshotManifest manifest, TableDescriptor tableDescriptor, org.apache.hadoop.fs.Path rootDir, ForeignExceptionDispatcher monitor, MonitoredTask status)
RestoreSnapshotHelper(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, SnapshotManifest manifest, TableDescriptor tableDescriptor, org.apache.hadoop.fs.Path rootDir, ForeignExceptionDispatcher monitor, MonitoredTask status, boolean createBackRefs)

Method Summary

Modifier and Type	Method and Description
private void	cloneHdfsMobRegion(Map<String,org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotRegionManifest> regionManifests, RegionInfo region) Clone the mob region.
private RegionInfo[]	cloneHdfsRegions(ThreadPoolExecutor exec, Map<String,org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotRegionManifest> regionManifests, List<RegionInfo> regions) Clone specified regions.
private void	cloneRegion(HRegion region, RegionInfo snapshotRegionInfo, org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotRegionManifest manifest) Clone region directory content from the snapshot info.
private void	cloneRegion(org.apache.hadoop.fs.Path regionDir, RegionInfo snapshotRegionInfo, org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotRegionManifest manifest) Clone region directory content from the snapshot info.
RegionInfo	cloneRegionInfo(RegionInfo snapshotRegionInfo) Create a new RegionInfo from the snapshot region info.
static RegionInfo	cloneRegionInfo(TableName tableName, RegionInfo snapshotRegionInfo)
static RestoreSnapshotHelper.RestoreMetaChanges	copySnapshotForScanner(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path rootDir, org.apache.hadoop.fs.Path restoreDir, String snapshotName) Copy the snapshot files for a snapshot scanner, discards meta changes.
private Map<String,List<org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotRegionManifest.StoreFile>>	getRegionHFileReferences(org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotRegionManifest manifest)
private Set<String>	getTableRegionFamilyFiles(org.apache.hadoop.fs.Path familyDir)
private List<RegionInfo>	getTableRegions()
private void	removeHdfsRegions(ThreadPoolExecutor exec, List<RegionInfo> regions) Remove specified regions from the file-system, using the archiver.
private void	restoreHdfsMobRegions(ThreadPoolExecutor exec, Map<String,org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotRegionManifest> regionManifests, List<RegionInfo> regions) Restore specified mob regions by restoring content to the snapshot state.
RestoreSnapshotHelper.RestoreMetaChanges	restoreHdfsRegions() Restore the on-disk table to a specified snapshot state.
private RestoreSnapshotHelper.RestoreMetaChanges	restoreHdfsRegions(ThreadPoolExecutor exec)
private void	restoreHdfsRegions(ThreadPoolExecutor exec, Map<String,org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotRegionManifest> regionManifests, List<RegionInfo> regions) Restore specified regions by restoring content to the snapshot state.
private void	restoreMobRegion(RegionInfo regionInfo, org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotRegionManifest regionManifest) Restore mob region by removing files not in the snapshot and adding the missing ones from the snapshot.
private void	restoreReferenceFile(org.apache.hadoop.fs.Path familyDir, RegionInfo regionInfo, org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotRegionManifest.StoreFile storeFile) Create a new Reference as copy of the source one.
private void	restoreRegion(RegionInfo regionInfo, org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotRegionManifest regionManifest) Restore region by removing files not in the snapshot and adding the missing ones from the snapshot.
private void	restoreRegion(RegionInfo regionInfo, org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotRegionManifest regionManifest, org.apache.hadoop.fs.Path regionDir) Restore region by removing files not in the snapshot and adding the missing ones from the snapshot.
static void	restoreSnapshotAcl(org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription snapshot, TableName newTableName, org.apache.hadoop.conf.Configuration conf)
private void	restoreStoreFile(org.apache.hadoop.fs.Path familyDir, RegionInfo regionInfo, org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotRegionManifest.StoreFile storeFile, boolean createBackRef) Create a new HFileLink to reference the store file.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

private static final org.slf4j.Logger LOG

regionsMap

private final Map<byte[],byte[]> regionsMap

parentsMap

private final Map<String,Pair<String,String>> parentsMap

monitor

private final ForeignExceptionDispatcher monitor

status

private final MonitoredTask status

snapshotManifest

private final SnapshotManifest snapshotManifest

snapshotDesc

private final org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription snapshotDesc

snapshotTable

private final TableName snapshotTable

tableDesc

private final TableDescriptor tableDesc

rootDir

private final org.apache.hadoop.fs.Path rootDir

tableDir

private final org.apache.hadoop.fs.Path tableDir

conf

private final org.apache.hadoop.conf.Configuration conf

fs

private final org.apache.hadoop.fs.FileSystem fs

createBackRefs

private final boolean createBackRefs

Constructor Detail

RestoreSnapshotHelper

public RestoreSnapshotHelper(org.apache.hadoop.conf.Configuration conf,
                             org.apache.hadoop.fs.FileSystem fs,
                             SnapshotManifest manifest,
                             TableDescriptor tableDescriptor,
                             org.apache.hadoop.fs.Path rootDir,
                             ForeignExceptionDispatcher monitor,
                             MonitoredTask status)

RestoreSnapshotHelper

public RestoreSnapshotHelper(org.apache.hadoop.conf.Configuration conf,
                             org.apache.hadoop.fs.FileSystem fs,
                             SnapshotManifest manifest,
                             TableDescriptor tableDescriptor,
                             org.apache.hadoop.fs.Path rootDir,
                             ForeignExceptionDispatcher monitor,
                             MonitoredTask status,
                             boolean createBackRefs)

Method Detail

restoreHdfsRegions

public RestoreSnapshotHelper.RestoreMetaChanges restoreHdfsRegions()
                                                            throws IOException

Restore the on-disk table to a specified snapshot state.

Returns:

the set of regions touched by the restore operation

Throws:

IOException

restoreHdfsRegions

private RestoreSnapshotHelper.RestoreMetaChanges restoreHdfsRegions(ThreadPoolExecutor exec)
                                                             throws IOException

Throws:

IOException

removeHdfsRegions

private void removeHdfsRegions(ThreadPoolExecutor exec,
                               List<RegionInfo> regions)
                        throws IOException

Remove specified regions from the file-system, using the archiver.

Throws:

IOException

restoreHdfsRegions

private void restoreHdfsRegions(ThreadPoolExecutor exec,
                                Map<String,org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotRegionManifest> regionManifests,
                                List<RegionInfo> regions)
                         throws IOException

Restore specified regions by restoring content to the snapshot state.

Throws:

IOException

restoreHdfsMobRegions

private void restoreHdfsMobRegions(ThreadPoolExecutor exec,
                                   Map<String,org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotRegionManifest> regionManifests,
                                   List<RegionInfo> regions)
                            throws IOException

Restore specified mob regions by restoring content to the snapshot state.

Throws:

IOException

getRegionHFileReferences

private Map<String,List<org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotRegionManifest.StoreFile>> getRegionHFileReferences(org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotRegionManifest manifest)

restoreRegion

private void restoreRegion(RegionInfo regionInfo,
                           org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotRegionManifest regionManifest)
                    throws IOException

Restore region by removing files not in the snapshot and adding the missing ones from the snapshot.

Throws:

IOException

restoreMobRegion

private void restoreMobRegion(RegionInfo regionInfo,
                              org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotRegionManifest regionManifest)
                       throws IOException

Restore mob region by removing files not in the snapshot and adding the missing ones from the snapshot.

Throws:

IOException

restoreRegion

private void restoreRegion(RegionInfo regionInfo,
                           org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotRegionManifest regionManifest,
                           org.apache.hadoop.fs.Path regionDir)
                    throws IOException

Restore region by removing files not in the snapshot and adding the missing ones from the snapshot.

Throws:

IOException

getTableRegionFamilyFiles

private Set<String> getTableRegionFamilyFiles(org.apache.hadoop.fs.Path familyDir)
                                       throws IOException

Returns:

The set of files in the specified family directory.

Throws:

IOException

cloneHdfsRegions

private RegionInfo[] cloneHdfsRegions(ThreadPoolExecutor exec,
                                      Map<String,org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotRegionManifest> regionManifests,
                                      List<RegionInfo> regions)
                               throws IOException

Clone specified regions. For each region create a new region and create a HFileLink for each hfile.

Throws:

IOException

cloneHdfsMobRegion

private void cloneHdfsMobRegion(Map<String,org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotRegionManifest> regionManifests,
                                RegionInfo region)
                         throws IOException

Clone the mob region. For the region create a new region and create a HFileLink for each hfile.

Throws:

IOException

cloneRegion

private void cloneRegion(org.apache.hadoop.fs.Path regionDir,
                         RegionInfo snapshotRegionInfo,
                         org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotRegionManifest manifest)
                  throws IOException

Clone region directory content from the snapshot info. Each region is encoded with the table name, so the cloned region will have a different region name. Instead of copying the hfiles a HFileLink is created.

Parameters:

regionDir - Path cloned dir

snapshotRegionInfo -

Throws:

IOException

cloneRegion

private void cloneRegion(HRegion region,
                         RegionInfo snapshotRegionInfo,
                         org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotRegionManifest manifest)
                  throws IOException

Clone region directory content from the snapshot info. Each region is encoded with the table name, so the cloned region will have a different region name. Instead of copying the hfiles a HFileLink is created.

Parameters:

region - HRegion cloned

snapshotRegionInfo -

Throws:

IOException

restoreStoreFile

private void restoreStoreFile(org.apache.hadoop.fs.Path familyDir,
                              RegionInfo regionInfo,
                              org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotRegionManifest.StoreFile storeFile,
                              boolean createBackRef)
                       throws IOException

Create a new HFileLink to reference the store file.

The store file in the snapshot can be a simple hfile, an HFileLink or a reference.

• hfile: abc -> table=region-abc
• reference: abc.1234 -> table=region-abc.1234
• hfilelink: table=region-hfile -> table=region-hfile

Parameters:

familyDir - destination directory for the store file

regionInfo - destination region info for the table

createBackRef - - Whether back reference should be created. Defaults to true.

storeFile - store file name (can be a Reference, HFileLink or simple HFile)

Throws:

IOException

restoreReferenceFile

private void restoreReferenceFile(org.apache.hadoop.fs.Path familyDir,
                                  RegionInfo regionInfo,
                                  org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotRegionManifest.StoreFile storeFile)
                           throws IOException

Create a new Reference as copy of the source one.

 The source table looks like:
    1234/abc      (original file)
    5678/abc.1234 (reference file)

 After the clone operation looks like:
   wxyz/table=1234-abc
   stuv/table=1234-abc.wxyz

 NOTE that the region name in the clone changes (md5 of regioninfo)
 and the reference should reflect that change.
 

Parameters:

familyDir - destination directory for the store file

regionInfo - destination region info for the table

storeFile - reference file name

Throws:

IOException

cloneRegionInfo

public RegionInfo cloneRegionInfo(RegionInfo snapshotRegionInfo)

Create a new RegionInfo from the snapshot region info. Keep the same startKey, endKey, regionId and split information but change the table name.

Parameters:

snapshotRegionInfo - Info for region to clone.

Returns:

the new HRegion instance

cloneRegionInfo

public static RegionInfo cloneRegionInfo(TableName tableName,
                                         RegionInfo snapshotRegionInfo)

getTableRegions

private List<RegionInfo> getTableRegions()
                                  throws IOException

Returns:

the set of the regions contained in the table

Throws:

IOException

copySnapshotForScanner

public static RestoreSnapshotHelper.RestoreMetaChanges copySnapshotForScanner(org.apache.hadoop.conf.Configuration conf,
                                                                              org.apache.hadoop.fs.FileSystem fs,
                                                                              org.apache.hadoop.fs.Path rootDir,
                                                                              org.apache.hadoop.fs.Path restoreDir,
                                                                              String snapshotName)
                                                                       throws IOException

Copy the snapshot files for a snapshot scanner, discards meta changes.

Parameters:

conf -

fs -

rootDir -

restoreDir -

snapshotName -

Throws:

IOException

restoreSnapshotAcl

public static void restoreSnapshotAcl(org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription snapshot,
                                      TableName newTableName,
                                      org.apache.hadoop.conf.Configuration conf)
                               throws IOException

Throws:

IOException