ExportSnapshot (Apache HBase 1.2.12 API)

java.lang.Object
- org.apache.hadoop.conf.Configured
- - org.apache.hadoop.hbase.snapshot.ExportSnapshot

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, org.apache.hadoop.util.Tool
```
@InterfaceAudience.Public
@InterfaceStability.Evolving
public class ExportSnapshot
extends org.apache.hadoop.conf.Configured
implements org.apache.hadoop.util.Tool
```
Export the specified snapshot to a given FileSystem. The .snapshot/name folder is copied to the destination cluster and then all the hfiles/wals are copied using a Map-Reduce Job in the .archive/ location. When everything is done, the second cluster can restore the snapshot.

Nested Class Summary

Nested Classes
Modifier and Type	Class and Description
`static class`	`ExportSnapshot.Counter`
`private static class`	`ExportSnapshot.ExportMapper`
`private static class`	`ExportSnapshot.ExportSnapshotInputFormat`

Field Summary

Fields
Modifier and Type	Field and Description
`private static String`	`CONF_BANDWIDTH_MB`
`private static String`	`CONF_BUFFER_SIZE`
`private static String`	`CONF_CHECKSUM_VERIFY`
`static String`	`CONF_DEST_PREFIX` Configuration prefix for overrides for the destination filesystem
`private static String`	`CONF_FILES_GROUP`
`private static String`	`CONF_FILES_MODE`
`private static String`	`CONF_FILES_USER`
`private static String`	`CONF_INPUT_ROOT`
`private static String`	`CONF_MAP_GROUP`
`private static String`	`CONF_NUM_SPLITS`
`private static String`	`CONF_OUTPUT_ROOT`
`protected static String`	`CONF_SKIP_TMP`
`private static String`	`CONF_SNAPSHOT_DIR`
`private static String`	`CONF_SNAPSHOT_NAME`
`static String`	`CONF_SOURCE_PREFIX` Configuration prefix for overrides for the source filesystem
`(package private) static String`	`CONF_TEST_FAILURE`
`(package private) static String`	`CONF_TEST_RETRY`
`private static String`	`INPUT_FOLDER_PREFIX`
`private static org.apache.commons.logging.Log`	`LOG`
`private static String`	`MR_NUM_MAPS`
`static String`	`NAME`

Constructor Summary

Constructors
Constructor and Description

ExportSnapshot()

Constructors
Constructor and Description
`ExportSnapshot()`

Method Summary

Methods
Modifier and Type	Method and Description
`(package private) static List<List<Pair<org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos.SnapshotFileInfo,Long>>>`	`getBalancedSplits(List<Pair<org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos.SnapshotFileInfo,Long>> files, int ngroups)` Given a list of file paths and sizes, create around ngroups in as balanced a way as possible.
`private static List<Pair<org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos.SnapshotFileInfo,Long>>`	`getSnapshotFiles(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path snapshotDir)` Extract the list of files (HFiles/WALs) to copy using Map-Reduce.
`(package private) static int`	`innerMain(org.apache.hadoop.conf.Configuration conf, String[] args)` The guts of the `main(java.lang.String[])` method.
`static void`	`main(String[] args)`
`private void`	`printUsageAndExit()`
`int`	`run(String[] args)` Execute the export snapshot by copying the snapshot metadata, hfiles and wals.
`private void`	`runCopyJob(org.apache.hadoop.fs.Path inputRoot, org.apache.hadoop.fs.Path outputRoot, String snapshotName, org.apache.hadoop.fs.Path snapshotDir, boolean verifyChecksum, String filesUser, String filesGroup, int filesMode, int mappers, int bandwidthMB)` Run Map-Reduce Job to perform the files copy.
`private void`	`setOwner(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, String user, String group, boolean recursive)` Set path ownership.
`private void`	`setPermission(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, short filesMode, boolean recursive)` Set path permission.
`private void`	`verifySnapshot(org.apache.hadoop.conf.Configuration baseConf, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path rootDir, org.apache.hadoop.fs.Path snapshotDir)`

Methods inherited from class org.apache.hadoop.conf.Configured
getConf, setConf

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.conf.Configurable
getConf, setConf

Field Detail
- NAME
```
public static final String NAME
```
  See Also:
  Constant Field Values
- CONF_SOURCE_PREFIX
```
public static final String CONF_SOURCE_PREFIX
```
  Configuration prefix for overrides for the source filesystem
  
  See Also:
  Constant Field Values
- CONF_DEST_PREFIX
```
public static final String CONF_DEST_PREFIX
```
  Configuration prefix for overrides for the destination filesystem
  
  See Also:
  Constant Field Values
- LOG
```
private static final org.apache.commons.logging.Log LOG
```
- MR_NUM_MAPS
```
private static final String MR_NUM_MAPS
```
  See Also:
  Constant Field Values
- CONF_NUM_SPLITS
```
private static final String CONF_NUM_SPLITS
```
  See Also:
  Constant Field Values
- CONF_SNAPSHOT_NAME
```
private static final String CONF_SNAPSHOT_NAME
```
  See Also:
  Constant Field Values
- CONF_SNAPSHOT_DIR
```
private static final String CONF_SNAPSHOT_DIR
```
  See Also:
  Constant Field Values
- CONF_FILES_USER
```
private static final String CONF_FILES_USER
```
  See Also:
  Constant Field Values
- CONF_FILES_GROUP
```
private static final String CONF_FILES_GROUP
```
  See Also:
  Constant Field Values
- CONF_FILES_MODE
```
private static final String CONF_FILES_MODE
```
  See Also:
  Constant Field Values
- CONF_CHECKSUM_VERIFY
```
private static final String CONF_CHECKSUM_VERIFY
```
  See Also:
  Constant Field Values
- CONF_OUTPUT_ROOT
```
private static final String CONF_OUTPUT_ROOT
```
  See Also:
  Constant Field Values
- CONF_INPUT_ROOT
```
private static final String CONF_INPUT_ROOT
```
  See Also:
  Constant Field Values
- CONF_BUFFER_SIZE
```
private static final String CONF_BUFFER_SIZE
```
  See Also:
  Constant Field Values
- CONF_MAP_GROUP
```
private static final String CONF_MAP_GROUP
```
  See Also:
  Constant Field Values
- CONF_BANDWIDTH_MB
```
private static final String CONF_BANDWIDTH_MB
```
  See Also:
  Constant Field Values
- CONF_SKIP_TMP
```
protected static final String CONF_SKIP_TMP
```
  See Also:
  Constant Field Values
- CONF_TEST_FAILURE
```
static final String CONF_TEST_FAILURE
```
  See Also:
  Constant Field Values
- CONF_TEST_RETRY
```
static final String CONF_TEST_RETRY
```
  See Also:
  Constant Field Values
- INPUT_FOLDER_PREFIX
```
private static final String INPUT_FOLDER_PREFIX
```
  See Also:
  Constant Field Values

Constructor Detail
- ExportSnapshot
```
public ExportSnapshot()
```

Method Detail

getSnapshotFiles

private static List<Pair<org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos.SnapshotFileInfo,Long>> getSnapshotFiles(org.apache.hadoop.conf.Configuration conf,
                                                                                                           org.apache.hadoop.fs.FileSystem fs,
                                                                                                           org.apache.hadoop.fs.Path snapshotDir)
                                                                                                                     throws IOException

Extract the list of files (HFiles/WALs) to copy using Map-Reduce.

Returns:: list of files referenced by the snapshot (pair of path and size)
Throws:: IOException

getBalancedSplits

static List<List<Pair<org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos.SnapshotFileInfo,Long>>> getBalancedSplits(List<Pair<org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos.SnapshotFileInfo,Long>> files,
                                                                                                                  int ngroups)

Given a list of file paths and sizes, create around ngroups in as balanced a way as possible. The groups created will have similar amounts of bytes.

The algorithm used is pretty straightforward; the file list is sorted by size, and then each group fetch the bigger file available, iterating through groups alternating the direction.

runCopyJob

private void runCopyJob(org.apache.hadoop.fs.Path inputRoot,
              org.apache.hadoop.fs.Path outputRoot,
              String snapshotName,
              org.apache.hadoop.fs.Path snapshotDir,
              boolean verifyChecksum,
              String filesUser,
              String filesGroup,
              int filesMode,
              int mappers,
              int bandwidthMB)
                 throws IOException,
                        InterruptedException,
                        ClassNotFoundException

Run Map-Reduce Job to perform the files copy.

Throws:: IOException; InterruptedException; ClassNotFoundException

verifySnapshot

private void verifySnapshot(org.apache.hadoop.conf.Configuration baseConf,
                  org.apache.hadoop.fs.FileSystem fs,
                  org.apache.hadoop.fs.Path rootDir,
                  org.apache.hadoop.fs.Path snapshotDir)
                     throws IOException

Throws:: IOException

setOwner

private void setOwner(org.apache.hadoop.fs.FileSystem fs,
            org.apache.hadoop.fs.Path path,
            String user,
            String group,
            boolean recursive)
               throws IOException

Set path ownership.

Throws:: IOException

setPermission

private void setPermission(org.apache.hadoop.fs.FileSystem fs,
                 org.apache.hadoop.fs.Path path,
                 short filesMode,
                 boolean recursive)
                    throws IOException

Set path permission.

Throws:: IOException

run
```
public int run(String[] args)
        throws IOException
```
Execute the export snapshot by copying the snapshot metadata, hfiles and wals.

Specified by:

run in interface org.apache.hadoop.util.Tool

Returns:
0 on success, and != 0 upon failure.

Throws:

IOException

printUsageAndExit
```
private void printUsageAndExit()
```

innerMain
```
static int innerMain(org.apache.hadoop.conf.Configuration conf,
            String[] args)
              throws Exception
```
The guts of the main(java.lang.String[]) method. Call this method to avoid the main(String[]) System.exit.

Parameters:
args -

Returns:
errCode

Throws:

Exception

main

public static void main(String[] args)
                 throws Exception

Throws:: Exception

Class ExportSnapshot

Nested Class Summary

Field Summary

Constructor Summary

Method Summary

Methods inherited from class org.apache.hadoop.conf.Configured

Methods inherited from class java.lang.Object

Methods inherited from interface org.apache.hadoop.conf.Configurable

Field Detail

NAME

CONF_SOURCE_PREFIX

CONF_DEST_PREFIX

LOG

MR_NUM_MAPS

CONF_NUM_SPLITS

CONF_SNAPSHOT_NAME

CONF_SNAPSHOT_DIR

CONF_FILES_USER

CONF_FILES_GROUP

CONF_FILES_MODE

CONF_CHECKSUM_VERIFY

CONF_OUTPUT_ROOT

CONF_INPUT_ROOT

CONF_BUFFER_SIZE

CONF_MAP_GROUP

CONF_BANDWIDTH_MB

CONF_SKIP_TMP

CONF_TEST_FAILURE

CONF_TEST_RETRY

INPUT_FOLDER_PREFIX

Constructor Detail

ExportSnapshot