org.apache.hadoop.hbase.snapshot

Class SnapshotDescriptionUtils

java.lang.Object

org.apache.hadoop.hbase.snapshot.SnapshotDescriptionUtils

@InterfaceAudience.Private
public final class SnapshotDescriptionUtils
extends Object

Utility class to help manage SnapshotDesriptions.

Snapshots are laid out on disk like this:

 /hbase/.snapshots
          /.tmp                <---- working directory
          /[snapshot name]     <----- completed snapshot
 

A completed snapshot named 'completed' then looks like (multiple regions, servers, files, etc. signified by '...' on the same directory depth).

 /hbase/.snapshots/completed
                   .snapshotinfo          <--- Description of the snapshot
                   .tableinfo             <--- Copy of the tableinfo
                    /.logs
                        /[server_name]
                            /... [log files]
                         ...
                   /[region name]           <---- All the region's information
                   .regioninfo              <---- Copy of the HRegionInfo
                      /[column family name]
                          /[hfile name]     <--- name of the hfile in the real region
                          ...
                      ...
                    ...
 

Utility methods in this class are useful for getting the correct locations for different parts of the snapshot, as well as moving completed snapshots into place (see completeSnapshot(org.apache.hadoop.fs.Path, org.apache.hadoop.fs.Path, org.apache.hadoop.fs.FileSystem, org.apache.hadoop.fs.FileSystem, org.apache.hadoop.conf.Configuration), and writing the SnapshotProtos.SnapshotDescription to the working snapshot directory.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	SnapshotDescriptionUtils.CompletedSnaphotDirectoriesFilter Filter that only accepts completed snapshot directories

Field Summary

Fields

Modifier and Type	Field and Description
static long	DEFAULT_MAX_WAIT_TIME By default, wait 300 seconds for a snapshot to complete
private static org.slf4j.Logger	LOG
static String	MASTER_SNAPSHOT_TIMEOUT_MILLIS
static long	NO_SNAPSHOT_START_TIME_SPECIFIED Default value if no start time is specified
private static long	NO_SNAPSHOT_TTL_SPECIFIED
static int	SNAPSHOT_LAYOUT_VERSION Version of the fs layout for a snapshot.
static int	SNAPSHOT_TIMEOUT_MILLIS_DEFAULT Deprecated. Use DEFAULT_MAX_WAIT_TIME instead.
static String	SNAPSHOT_TIMEOUT_MILLIS_KEY Deprecated. Use MASTER_SNAPSHOT_TIMEOUT_MILLIS instead.
static String	SNAPSHOT_TMP_DIR_NAME Temporary directory under the snapshot directory to store in-progress snapshots
static String	SNAPSHOT_WORKING_DIR The configuration property that determines the filepath of the snapshot base working directory
static String	SNAPSHOTINFO_FILE The file contains the snapshot basic information and it is under the directory of a snapshot.

Constructor Summary

Constructors

Modifier	Constructor and Description
private	SnapshotDescriptionUtils()

Method Summary

Modifier and Type	Method and Description
static void	completeSnapshot(org.apache.hadoop.fs.Path snapshotDir, org.apache.hadoop.fs.Path workingDir, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.FileSystem workingDirFs, org.apache.hadoop.conf.Configuration conf) Commits the snapshot process by moving the working snapshot to the finalized filepath
static org.apache.hadoop.fs.Path	getCompletedSnapshotDir(org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription snapshot, org.apache.hadoop.fs.Path rootDir) Get the directory for a specified snapshot.
static org.apache.hadoop.fs.Path	getCompletedSnapshotDir(String snapshotName, org.apache.hadoop.fs.Path rootDir) Get the directory for a completed snapshot.
private static org.apache.hadoop.fs.Path	getDefaultWorkingSnapshotDir(org.apache.hadoop.fs.Path rootDir) Get the default working directory for snapshots - where they are built, where they are temporarily copied on export, etc.
static long	getMaxMasterTimeout(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription.Type type, long defaultMaxWaitTime)
static org.apache.hadoop.fs.Path	getSnapshotRootDir(org.apache.hadoop.fs.Path rootDir) Get the snapshot root directory.
static org.apache.hadoop.fs.Path	getSnapshotsDir(org.apache.hadoop.fs.Path rootDir)
private static org.apache.hadoop.fs.Path	getSpecifiedSnapshotDir(org.apache.hadoop.fs.Path snapshotsDir, String snapshotName) Get the directory within the given filepath to store the snapshot instance
static org.apache.hadoop.fs.Path	getWorkingSnapshotDir(org.apache.hadoop.fs.Path rootDir, org.apache.hadoop.conf.Configuration conf) Get the general working directory for snapshots - where they are built, where they are temporarily copied on export, etc.
static org.apache.hadoop.fs.Path	getWorkingSnapshotDir(org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription snapshot, org.apache.hadoop.fs.Path rootDir, org.apache.hadoop.conf.Configuration conf) Get the directory to build a snapshot, before it is finalized
static org.apache.hadoop.fs.Path	getWorkingSnapshotDir(String snapshotName, org.apache.hadoop.fs.Path rootDir, org.apache.hadoop.conf.Configuration conf) Get the directory to build a snapshot, before it is finalized
static boolean	isSecurityAvailable(org.apache.hadoop.conf.Configuration conf)
static boolean	isSnapshotOwner(SnapshotDescription snapshot, User user) Check if the user is this table snapshot's owner
static boolean	isSubDirectoryOf(org.apache.hadoop.fs.Path workingDir, org.apache.hadoop.fs.Path rootDir) Determines if the given workingDir is a subdirectory of the given "root directory"
static boolean	isWithinDefaultWorkingDir(org.apache.hadoop.fs.Path workingDir, org.apache.hadoop.conf.Configuration conf) Determines if the given workingDir is a subdirectory of the default working snapshot directory
static org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription	readSnapshotInfo(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path snapshotDir) Read in the HBaseProtos.SnapshotDescription stored for the snapshot in the passed directory
static org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription	validate(org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription snapshot, org.apache.hadoop.conf.Configuration conf) Convert the passed snapshot description into a 'full' snapshot description based on default parameters, if none have been supplied.
private static org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription	writeAclToSnapshotDescription(org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription snapshot, org.apache.hadoop.conf.Configuration conf)
static void	writeSnapshotInfo(org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription snapshot, org.apache.hadoop.fs.Path workingDir, org.apache.hadoop.fs.FileSystem fs) Write the snapshot description into the working directory of a snapshot

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

private static final org.slf4j.Logger LOG

SNAPSHOT_LAYOUT_VERSION

public static final int SNAPSHOT_LAYOUT_VERSION

Version of the fs layout for a snapshot. Future snapshots may have different file layouts, which we may need to read in differently.

See Also:

Constant Field Values

SNAPSHOTINFO_FILE

public static final String SNAPSHOTINFO_FILE

The file contains the snapshot basic information and it is under the directory of a snapshot.

See Also:

Constant Field Values

SNAPSHOT_TMP_DIR_NAME

public static final String SNAPSHOT_TMP_DIR_NAME

Temporary directory under the snapshot directory to store in-progress snapshots

See Also:

Constant Field Values

SNAPSHOT_WORKING_DIR

public static final String SNAPSHOT_WORKING_DIR

The configuration property that determines the filepath of the snapshot base working directory

See Also:

Constant Field Values

NO_SNAPSHOT_START_TIME_SPECIFIED

public static final long NO_SNAPSHOT_START_TIME_SPECIFIED

Default value if no start time is specified

See Also:

Constant Field Values

NO_SNAPSHOT_TTL_SPECIFIED

private static final long NO_SNAPSHOT_TTL_SPECIFIED

See Also:

Constant Field Values

MASTER_SNAPSHOT_TIMEOUT_MILLIS

public static final String MASTER_SNAPSHOT_TIMEOUT_MILLIS

See Also:

Constant Field Values

DEFAULT_MAX_WAIT_TIME

public static final long DEFAULT_MAX_WAIT_TIME

By default, wait 300 seconds for a snapshot to complete

See Also:

Constant Field Values

SNAPSHOT_TIMEOUT_MILLIS_DEFAULT

@Deprecated
public static final int SNAPSHOT_TIMEOUT_MILLIS_DEFAULT

Deprecated. Use DEFAULT_MAX_WAIT_TIME instead.

By default, check to see if the snapshot is complete (ms)

See Also:

Constant Field Values

SNAPSHOT_TIMEOUT_MILLIS_KEY

@Deprecated
public static final String SNAPSHOT_TIMEOUT_MILLIS_KEY

Deprecated. Use MASTER_SNAPSHOT_TIMEOUT_MILLIS instead.

Conf key for # of ms elapsed before injecting a snapshot timeout error when waiting for completion.

See Also:

Constant Field Values

Constructor Detail

SnapshotDescriptionUtils

private SnapshotDescriptionUtils()

Method Detail

getMaxMasterTimeout

public static long getMaxMasterTimeout(org.apache.hadoop.conf.Configuration conf,
                                       org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription.Type type,
                                       long defaultMaxWaitTime)

Parameters:

conf - Configuration from which to check for the timeout

type - type of snapshot being taken

defaultMaxWaitTime - Default amount of time to wait, if none is in the configuration

Returns:

the max amount of time the master should wait for a snapshot to complete

getSnapshotRootDir

public static org.apache.hadoop.fs.Path getSnapshotRootDir(org.apache.hadoop.fs.Path rootDir)

Get the snapshot root directory. All the snapshots are kept under this directory, i.e. ${hbase.rootdir}/.snapshot

Parameters:

rootDir - hbase root directory

Returns:

the base directory in which all snapshots are kept

getCompletedSnapshotDir

public static org.apache.hadoop.fs.Path getCompletedSnapshotDir(org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription snapshot,
                                                                org.apache.hadoop.fs.Path rootDir)

Get the directory for a specified snapshot. This directory is a sub-directory of snapshot root directory and all the data files for a snapshot are kept under this directory.

Parameters:

snapshot - snapshot being taken

rootDir - hbase root directory

Returns:

the final directory for the completed snapshot

getCompletedSnapshotDir

public static org.apache.hadoop.fs.Path getCompletedSnapshotDir(String snapshotName,
                                                                org.apache.hadoop.fs.Path rootDir)

Get the directory for a completed snapshot. This directory is a sub-directory of snapshot root directory and all the data files for a snapshot are kept under this directory.

Parameters:

snapshotName - name of the snapshot being taken

rootDir - hbase root directory

Returns:

the final directory for the completed snapshot

getWorkingSnapshotDir

public static org.apache.hadoop.fs.Path getWorkingSnapshotDir(org.apache.hadoop.fs.Path rootDir,
                                                              org.apache.hadoop.conf.Configuration conf)

Get the general working directory for snapshots - where they are built, where they are temporarily copied on export, etc.

Parameters:

rootDir - root directory of the HBase installation

conf - Configuration of the HBase instance

Returns:

Path to the snapshot tmp directory, relative to the passed root directory

getWorkingSnapshotDir

public static org.apache.hadoop.fs.Path getWorkingSnapshotDir(org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription snapshot,
                                                              org.apache.hadoop.fs.Path rootDir,
                                                              org.apache.hadoop.conf.Configuration conf)

Get the directory to build a snapshot, before it is finalized

Parameters:

snapshot - snapshot that will be built

rootDir - root directory of the hbase installation

conf - Configuration of the HBase instance

Returns:

Path where one can build a snapshot

getWorkingSnapshotDir

public static org.apache.hadoop.fs.Path getWorkingSnapshotDir(String snapshotName,
                                                              org.apache.hadoop.fs.Path rootDir,
                                                              org.apache.hadoop.conf.Configuration conf)

Get the directory to build a snapshot, before it is finalized

Parameters:

snapshotName - name of the snapshot

rootDir - root directory of the hbase installation

conf - Configuration of the HBase instance

Returns:

Path where one can build a snapshot

getSpecifiedSnapshotDir

private static final org.apache.hadoop.fs.Path getSpecifiedSnapshotDir(org.apache.hadoop.fs.Path snapshotsDir,
                                                                       String snapshotName)

Get the directory within the given filepath to store the snapshot instance

Parameters:

snapshotsDir - directory to store snapshot directory within

snapshotName - name of the snapshot to take

Returns:

the final directory for the snapshot in the given filepath

getSnapshotsDir

public static final org.apache.hadoop.fs.Path getSnapshotsDir(org.apache.hadoop.fs.Path rootDir)

Parameters:

rootDir - hbase root directory

Returns:

the directory for all completed snapshots;

isSubDirectoryOf

public static boolean isSubDirectoryOf(org.apache.hadoop.fs.Path workingDir,
                                       org.apache.hadoop.fs.Path rootDir)

Determines if the given workingDir is a subdirectory of the given "root directory"

Parameters:

workingDir - a directory to check

rootDir - root directory of the HBase installation

Returns:

true if the given workingDir is a subdirectory of the given root directory, false otherwise

isWithinDefaultWorkingDir

public static boolean isWithinDefaultWorkingDir(org.apache.hadoop.fs.Path workingDir,
                                                org.apache.hadoop.conf.Configuration conf)
                                         throws IOException

Determines if the given workingDir is a subdirectory of the default working snapshot directory

Parameters:

workingDir - a directory to check

conf - configuration for the HBase cluster

Returns:

true if the given workingDir is a subdirectory of the default working directory for snapshots, false otherwise

Throws:

IOException - if we can't get the root dir

getDefaultWorkingSnapshotDir

private static org.apache.hadoop.fs.Path getDefaultWorkingSnapshotDir(org.apache.hadoop.fs.Path rootDir)

Get the default working directory for snapshots - where they are built, where they are temporarily copied on export, etc.

Parameters:

rootDir - root directory of the HBase installation

Returns:

Path to the default snapshot tmp directory, relative to the passed root directory

validate

public static org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription validate(org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription snapshot,
                                                                                                            org.apache.hadoop.conf.Configuration conf)
                                                                                                     throws IllegalArgumentException,
                                                                                                            IOException

Convert the passed snapshot description into a 'full' snapshot description based on default parameters, if none have been supplied. This resolves any 'optional' parameters that aren't supplied to their default values.

Parameters:

snapshot - general snapshot descriptor

conf - Configuration to read configured snapshot defaults if snapshot is not complete

Returns:

a valid snapshot description

Throws:

IllegalArgumentException - if the SnapshotProtos.SnapshotDescription is not a complete SnapshotProtos.SnapshotDescription.

IOException

writeSnapshotInfo

public static void writeSnapshotInfo(org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription snapshot,
                                     org.apache.hadoop.fs.Path workingDir,
                                     org.apache.hadoop.fs.FileSystem fs)
                              throws IOException

Write the snapshot description into the working directory of a snapshot

Parameters:

snapshot - description of the snapshot being taken

workingDir - working directory of the snapshot

fs - FileSystem on which the snapshot should be taken

Throws:

IOException - if we can't reach the filesystem and the file cannot be cleaned up on failure

readSnapshotInfo

public static org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription readSnapshotInfo(org.apache.hadoop.fs.FileSystem fs,
                                                                                                                    org.apache.hadoop.fs.Path snapshotDir)
                                                                                                             throws CorruptedSnapshotException

Read in the HBaseProtos.SnapshotDescription stored for the snapshot in the passed directory

Parameters:

fs - filesystem where the snapshot was taken

snapshotDir - directory where the snapshot was stored

Returns:

the stored snapshot description

Throws:

CorruptedSnapshotException - if the snapshot cannot be read

completeSnapshot

public static void completeSnapshot(org.apache.hadoop.fs.Path snapshotDir,
                                    org.apache.hadoop.fs.Path workingDir,
                                    org.apache.hadoop.fs.FileSystem fs,
                                    org.apache.hadoop.fs.FileSystem workingDirFs,
                                    org.apache.hadoop.conf.Configuration conf)
                             throws SnapshotCreationException,
                                    IOException

Commits the snapshot process by moving the working snapshot to the finalized filepath

Parameters:

snapshotDir - The file path of the completed snapshots

workingDir - The file path of the in progress snapshots

fs - The file system of the completed snapshots

workingDirFs - The file system of the in progress snapshots

conf - Configuration

Throws:

SnapshotCreationException - if the snapshot could not be moved

IOException - the filesystem could not be reached

isSnapshotOwner

public static boolean isSnapshotOwner(SnapshotDescription snapshot,
                                      User user)

Check if the user is this table snapshot's owner

Parameters:

snapshot - the table snapshot description

user - the user

Returns:

true if the user is the owner of the snapshot, false otherwise or the snapshot owner field is not present.

isSecurityAvailable

public static boolean isSecurityAvailable(org.apache.hadoop.conf.Configuration conf)
                                   throws IOException

Throws:

IOException

writeAclToSnapshotDescription

private static org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription writeAclToSnapshotDescription(org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription snapshot,
                                                                                                                                  org.apache.hadoop.conf.Configuration conf)
                                                                                                                           throws IOException

Throws:

IOException