org.apache.hadoop.hbase.snapshot

Class ExportSnapshot

java.lang.Object

org.apache.hadoop.hbase.util.AbstractHBaseTool

org.apache.hadoop.hbase.snapshot.ExportSnapshot

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, org.apache.hadoop.util.Tool

@InterfaceAudience.Public
public class ExportSnapshot
extends AbstractHBaseTool
implements org.apache.hadoop.util.Tool

Export the specified snapshot to a given FileSystem. The .snapshot/name folder is copied to the destination cluster and then all the hfiles/wals are copied using a Map-Reduce Job in the .archive/ location. When everything is done, the second cluster can restore the snapshot.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	ExportSnapshot.Counter
private static class	ExportSnapshot.ExportMapper
private static class	ExportSnapshot.ExportSnapshotInputFormat
(package private) static class	ExportSnapshot.Options
(package private) static class	ExportSnapshot.Testing

Field Summary

Fields

Modifier and Type	Field and Description
private int	bandwidthMB
private static String	CONF_BANDWIDTH_MB
private static String	CONF_BUFFER_SIZE
private static String	CONF_CHECKSUM_VERIFY
private static String	CONF_COPY_MANIFEST_THREADS
static String	CONF_DEST_PREFIX Configuration prefix for overrides for the destination filesystem
private static String	CONF_FILES_GROUP
private static String	CONF_FILES_MODE
private static String	CONF_FILES_USER
private static String	CONF_INPUT_ROOT
private static String	CONF_MAP_GROUP
private static String	CONF_MR_JOB_NAME
private static String	CONF_NUM_SPLITS
private static String	CONF_OUTPUT_ROOT
protected static String	CONF_SKIP_TMP
private static String	CONF_SNAPSHOT_DIR
private static String	CONF_SNAPSHOT_NAME
static String	CONF_SOURCE_PREFIX Configuration prefix for overrides for the source filesystem
private static int	DEFAULT_COPY_MANIFEST_THREADS
private String	filesGroup
private int	filesMode
private String	filesUser
private org.apache.hadoop.fs.Path	inputRoot
private static org.slf4j.Logger	LOG
private int	mappers
private static String	MR_NUM_MAPS
static String	NAME
private org.apache.hadoop.fs.Path	outputRoot
private boolean	overwrite
private String	snapshotName
private String	targetName
private boolean	verifyChecksum
private boolean	verifyTarget

Fields inherited from class org.apache.hadoop.hbase.util.AbstractHBaseTool

cmdLineArgs, conf, EXIT_FAILURE, EXIT_SUCCESS, LONG_HELP_OPTION, options, SHORT_HELP_OPTION

Constructor Summary

Constructors

Constructor and Description
ExportSnapshot()

Method Summary

Modifier and Type	Method and Description
protected void	addOptions() Override this to add command-line options using AbstractHBaseTool.addOptWithArg(java.lang.String, java.lang.String) and similar methods.
int	doWork() Execute the export snapshot by copying the snapshot metadata, hfiles and wals.
(package private) static List<List<Pair<org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotFileInfo,Long>>>	getBalancedSplits(List<Pair<org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotFileInfo,Long>> files, int ngroups) Given a list of file paths and sizes, create around ngroups in as balanced a way as possible.
private static List<Pair<org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotFileInfo,Long>>	getSnapshotFiles(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path snapshotDir) Extract the list of files (HFiles/WALs) to copy using Map-Reduce.
static void	main(String[] args)
protected void	printUsage()
protected void	processOptions(org.apache.hbase.thirdparty.org.apache.commons.cli.CommandLine cmd) This method is called to process the options after they have been parsed.
private void	runCopyJob(org.apache.hadoop.fs.Path inputRoot, org.apache.hadoop.fs.Path outputRoot, String snapshotName, org.apache.hadoop.fs.Path snapshotDir, boolean verifyChecksum, String filesUser, String filesGroup, int filesMode, int mappers, int bandwidthMB) Run Map-Reduce Job to perform the files copy.
private void	setConfigParallel(org.apache.hadoop.fs.FileSystem outputFs, List<org.apache.hadoop.fs.Path> traversedPath, BiConsumer<org.apache.hadoop.fs.FileSystem,org.apache.hadoop.fs.Path> task, org.apache.hadoop.conf.Configuration conf)
private void	setOwnerParallel(org.apache.hadoop.fs.FileSystem outputFs, String filesUser, String filesGroup, org.apache.hadoop.conf.Configuration conf, List<org.apache.hadoop.fs.Path> traversedPath)
private void	setPermissionParallel(org.apache.hadoop.fs.FileSystem outputFs, short filesMode, List<org.apache.hadoop.fs.Path> traversedPath, org.apache.hadoop.conf.Configuration conf)
private void	verifySnapshot(org.apache.hadoop.conf.Configuration baseConf, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path rootDir, org.apache.hadoop.fs.Path snapshotDir)

Methods inherited from class org.apache.hadoop.hbase.util.AbstractHBaseTool

addOption, addOptNoArg, addOptNoArg, addOptWithArg, addOptWithArg, addRequiredOption, addRequiredOptWithArg, addRequiredOptWithArg, doStaticMain, getConf, getOptionAsDouble, getOptionAsInt, getOptionAsLong, newParser, parseArgs, parseInt, parseLong, printUsage, processOldArgs, run, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.util.Tool

run

Methods inherited from interface org.apache.hadoop.conf.Configurable

getConf, setConf

Field Detail

NAME

public static final String NAME

See Also:

Constant Field Values

CONF_SOURCE_PREFIX

public static final String CONF_SOURCE_PREFIX

Configuration prefix for overrides for the source filesystem

See Also:

Constant Field Values

CONF_DEST_PREFIX

public static final String CONF_DEST_PREFIX

Configuration prefix for overrides for the destination filesystem

See Also:

Constant Field Values

LOG

private static final org.slf4j.Logger LOG

MR_NUM_MAPS

private static final String MR_NUM_MAPS

See Also:

Constant Field Values

CONF_NUM_SPLITS

private static final String CONF_NUM_SPLITS

See Also:

Constant Field Values

CONF_SNAPSHOT_NAME

private static final String CONF_SNAPSHOT_NAME

See Also:

Constant Field Values

CONF_SNAPSHOT_DIR

private static final String CONF_SNAPSHOT_DIR

See Also:

Constant Field Values

CONF_FILES_USER

private static final String CONF_FILES_USER

See Also:

Constant Field Values

CONF_FILES_GROUP

private static final String CONF_FILES_GROUP

See Also:

Constant Field Values

CONF_FILES_MODE

private static final String CONF_FILES_MODE

See Also:

Constant Field Values

CONF_CHECKSUM_VERIFY

private static final String CONF_CHECKSUM_VERIFY

See Also:

Constant Field Values

CONF_OUTPUT_ROOT

private static final String CONF_OUTPUT_ROOT

See Also:

Constant Field Values

CONF_INPUT_ROOT

private static final String CONF_INPUT_ROOT

See Also:

Constant Field Values

CONF_BUFFER_SIZE

private static final String CONF_BUFFER_SIZE

See Also:

Constant Field Values

CONF_MAP_GROUP

private static final String CONF_MAP_GROUP

See Also:

Constant Field Values

CONF_BANDWIDTH_MB

private static final String CONF_BANDWIDTH_MB

See Also:

Constant Field Values

CONF_MR_JOB_NAME

private static final String CONF_MR_JOB_NAME

See Also:

Constant Field Values

CONF_SKIP_TMP

protected static final String CONF_SKIP_TMP

See Also:

Constant Field Values

CONF_COPY_MANIFEST_THREADS

private static final String CONF_COPY_MANIFEST_THREADS

See Also:

Constant Field Values

DEFAULT_COPY_MANIFEST_THREADS

private static final int DEFAULT_COPY_MANIFEST_THREADS

verifyTarget

private boolean verifyTarget

verifyChecksum

private boolean verifyChecksum

snapshotName

private String snapshotName

targetName

private String targetName

overwrite

private boolean overwrite

filesGroup

private String filesGroup

filesUser

private String filesUser

outputRoot

private org.apache.hadoop.fs.Path outputRoot

inputRoot

private org.apache.hadoop.fs.Path inputRoot

bandwidthMB

private int bandwidthMB

filesMode

private int filesMode

mappers

private int mappers

Constructor Detail

ExportSnapshot

public ExportSnapshot()

Method Detail

getSnapshotFiles

private static List<Pair<org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotFileInfo,Long>> getSnapshotFiles(org.apache.hadoop.conf.Configuration conf,
                                                                                                                                   org.apache.hadoop.fs.FileSystem fs,
                                                                                                                                   org.apache.hadoop.fs.Path snapshotDir)
                                                                                                                            throws IOException

Extract the list of files (HFiles/WALs) to copy using Map-Reduce.

Returns:

list of files referenced by the snapshot (pair of path and size)

Throws:

IOException

getBalancedSplits

static List<List<Pair<org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotFileInfo,Long>>> getBalancedSplits(List<Pair<org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotFileInfo,Long>> files,
                                                                                                                                  int ngroups)

Given a list of file paths and sizes, create around ngroups in as balanced a way as possible. The groups created will have similar amounts of bytes.

The algorithm used is pretty straightforward; the file list is sorted by size, and then each group fetch the bigger file available, iterating through groups alternating the direction.

runCopyJob

private void runCopyJob(org.apache.hadoop.fs.Path inputRoot,
                        org.apache.hadoop.fs.Path outputRoot,
                        String snapshotName,
                        org.apache.hadoop.fs.Path snapshotDir,
                        boolean verifyChecksum,
                        String filesUser,
                        String filesGroup,
                        int filesMode,
                        int mappers,
                        int bandwidthMB)
                 throws IOException,
                        InterruptedException,
                        ClassNotFoundException

Run Map-Reduce Job to perform the files copy.

Throws:

IOException

InterruptedException

ClassNotFoundException

verifySnapshot

private void verifySnapshot(org.apache.hadoop.conf.Configuration baseConf,
                            org.apache.hadoop.fs.FileSystem fs,
                            org.apache.hadoop.fs.Path rootDir,
                            org.apache.hadoop.fs.Path snapshotDir)
                     throws IOException

Throws:

IOException

setConfigParallel

private void setConfigParallel(org.apache.hadoop.fs.FileSystem outputFs,
                               List<org.apache.hadoop.fs.Path> traversedPath,
                               BiConsumer<org.apache.hadoop.fs.FileSystem,org.apache.hadoop.fs.Path> task,
                               org.apache.hadoop.conf.Configuration conf)
                        throws IOException

Throws:

IOException

setOwnerParallel

private void setOwnerParallel(org.apache.hadoop.fs.FileSystem outputFs,
                              String filesUser,
                              String filesGroup,
                              org.apache.hadoop.conf.Configuration conf,
                              List<org.apache.hadoop.fs.Path> traversedPath)
                       throws IOException

Throws:

IOException

setPermissionParallel

private void setPermissionParallel(org.apache.hadoop.fs.FileSystem outputFs,
                                   short filesMode,
                                   List<org.apache.hadoop.fs.Path> traversedPath,
                                   org.apache.hadoop.conf.Configuration conf)
                            throws IOException

Throws:

IOException

processOptions

protected void processOptions(org.apache.hbase.thirdparty.org.apache.commons.cli.CommandLine cmd)

Description copied from class: AbstractHBaseTool

This method is called to process the options after they have been parsed.

Specified by:

processOptions in class AbstractHBaseTool

doWork

public int doWork()
           throws IOException

Execute the export snapshot by copying the snapshot metadata, hfiles and wals.

Specified by:

doWork in class AbstractHBaseTool

Returns:

0 on success, and != 0 upon failure.

Throws:

IOException

printUsage

protected void printUsage()

Overrides:

printUsage in class AbstractHBaseTool

addOptions

protected void addOptions()

Description copied from class: AbstractHBaseTool

Override this to add command-line options using AbstractHBaseTool.addOptWithArg(java.lang.String, java.lang.String) and similar methods.

Specified by:

addOptions in class AbstractHBaseTool

main

public static void main(String[] args)