Package org.apache.hadoop.hbase.backup.impl

Class IncrementalTableBackupClient

java.lang.Object

org.apache.hadoop.hbase.backup.impl.TableBackupClient

org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient

@Private
public class IncrementalTableBackupClient
extends TableBackupClient

Incremental backup implementation. See the execute method.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.hbase.backup.impl.TableBackupClient

TableBackupClient.Stage

Field Summary

Fields

Modifier and Type	Field	Description
private static final String	BULKLOAD_COLLECTOR_OUTPUT
private static final org.slf4j.Logger	LOG

Fields inherited from class org.apache.hadoop.hbase.backup.impl.TableBackupClient

BACKUP_CLIENT_IMPL_CLASS, BACKUP_TEST_MODE_STAGE, backupId, backupInfo, backupManager, conf, conn, fs, newTimestamps, tableList

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	IncrementalTableBackupClient()
	IncrementalTableBackupClient(Connection conn, String backupId, BackupRequest request)

Method Summary

Modifier and Type	Method	Description
private static boolean	areCfsCompatible(ColumnFamilyDescriptor[] currentCfs, ColumnFamilyDescriptor[] backupCfs)
protected void	convertWALsToHFiles(Map<TableName,List<String>> tablesToWALFileList, Map<TableName,Long> tablesToPrevBackupTs)
protected void	deleteBulkLoadDirectory()
private void	deleteDirectory(org.apache.hadoop.fs.Path path)
protected void	deleteTmpBackupDirectory()
void	execute()	Backup request execution.
protected List<String>	filterMissingFiles(List<String> incrBackupFileList)
protected org.apache.hadoop.fs.Path	getBulkOutputDir()	Creates a path to a temporary backup directory.
protected org.apache.hadoop.fs.Path	getBulkOutputDirForTable(TableName table)	Creates a path to the bulk load output directory for a table.
private Map<TableName,String>	getFullBackupIds()
protected static int	getIndex(TableName tbl, List<TableName> sTableList)
private org.apache.hadoop.fs.Path	getTargetDirForTable(TableName table)	Creates a path to a destination directory for bulk loaded HFiles.
protected org.apache.hadoop.fs.Path	getTmpBackupDir()	Creates a path to the backup root directory's temporary subdirectory.
protected org.apache.hadoop.fs.Path	getTmpBackupDirForTable(TableName table)	Creates a path to a table's directory within the temporary directory.
protected List<BulkLoad>	handleBulkLoad(List<TableName> tablesToBackup, Map<TableName,List<String>> tablesToWALFileList, Map<TableName,Long> tablesToPrevBackupTs)	Reads bulk load records from backup table, iterates through the records and forms the paths for bulk loaded hfiles.
private void	incrementalCopyBulkloadHFiles(org.apache.hadoop.fs.FileSystem tgtFs, TableName tn)
protected void	incrementalCopyHFiles(String[] files, String backupDest)
protected boolean	isActiveWalPath(org.apache.hadoop.fs.Path p)	Check if a given path belongs to active WAL directory
private void	mergeSplitAndCopyBulkloadedHFiles(List<String> activeFiles, List<String> archiveFiles, TableName tn, org.apache.hadoop.fs.FileSystem tgtFs)
private void	mergeSplitAndCopyBulkloadedHFiles(List<String> files, TableName tn, org.apache.hadoop.fs.FileSystem tgtFs)
private void	setBulkOutputPath(org.apache.hadoop.conf.Configuration conf, List<String> tableList)
private void	setupRegionLocator()
protected boolean	tableExists(TableName table, Connection conn)
void	updateFileLists(List<String> activeFiles, List<String> archiveFiles)
private void	verifyCfCompatibility(Set<TableName> tables, Map<TableName,String> tablesToFullBackupId)	Verifies that the current table descriptor CFs matches the descriptor CFs of the last full backup for the tables.
protected void	walToHFiles(List<String> dirPaths, List<String> tableList, long previousBackupTs)

Methods inherited from class org.apache.hadoop.hbase.backup.impl.TableBackupClient

addManifest, beginBackup, cleanupAndRestoreBackupSystem, cleanupExportSnapshotLog, cleanupTargetDir, completeBackup, deleteSnapshots, failBackup, failStageIf, getAncestors, getMessage, getTestStage, init, obtainBackupMetaDataStr

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

LOG

private static final org.slf4j.Logger LOG

BULKLOAD_COLLECTOR_OUTPUT

private static final String BULKLOAD_COLLECTOR_OUTPUT

See Also:

• Constant Field Values

Constructor Details

IncrementalTableBackupClient

protected IncrementalTableBackupClient()

IncrementalTableBackupClient

public IncrementalTableBackupClient(Connection conn,
 String backupId,
 BackupRequest request)
                             throws IOException

Throws:

IOException

Method Details

filterMissingFiles

protected List<String> filterMissingFiles(List<String> incrBackupFileList)
                                   throws IOException

Throws:

IOException

isActiveWalPath

protected boolean isActiveWalPath(org.apache.hadoop.fs.Path p)

Check if a given path belongs to active WAL directory

Parameters:

p - path

Returns:

true, if yes

getIndex

protected static int getIndex(TableName tbl,
 List<TableName> sTableList)

handleBulkLoad

protected List<BulkLoad> handleBulkLoad(List<TableName> tablesToBackup,
 Map<TableName,List<String>> tablesToWALFileList,
 Map<TableName,Long> tablesToPrevBackupTs)
                                 throws IOException

Reads bulk load records from backup table, iterates through the records and forms the paths for bulk loaded hfiles. Copies the bulk loaded hfiles to the backup destination. This method does NOT clean up the entries in the bulk load system table. Those entries should not be cleaned until the backup is marked as complete.

Parameters:

tablesToBackup - list of tables to be backed up

Throws:

IOException

mergeSplitAndCopyBulkloadedHFiles

private void mergeSplitAndCopyBulkloadedHFiles(List<String> activeFiles,
 List<String> archiveFiles,
 TableName tn,
 org.apache.hadoop.fs.FileSystem tgtFs)
                                        throws IOException

Throws:

IOException

mergeSplitAndCopyBulkloadedHFiles

private void mergeSplitAndCopyBulkloadedHFiles(List<String> files,
 TableName tn,
 org.apache.hadoop.fs.FileSystem tgtFs)
                                        throws IOException

Throws:

IOException

updateFileLists

public void updateFileLists(List<String> activeFiles,
 List<String> archiveFiles)
                     throws IOException

Throws:

IOException

execute

public void execute()
             throws IOException,
ColumnFamilyMismatchException

Description copied from class: TableBackupClient

Backup request execution.

Specified by:

execute in class TableBackupClient

Throws:

IOException - If the execution of the backup fails

ColumnFamilyMismatchException - If the column families of the current table do not match the column families for the last full backup. In which case, a full backup should be taken

incrementalCopyHFiles

protected void incrementalCopyHFiles(String[] files,
 String backupDest)
                              throws IOException

Throws:

IOException

deleteBulkLoadDirectory

protected void deleteBulkLoadDirectory()
                                throws IOException

Throws:

IOException

deleteTmpBackupDirectory

protected void deleteTmpBackupDirectory()
                                 throws IOException

Throws:

IOException

deleteDirectory

private void deleteDirectory(org.apache.hadoop.fs.Path path)
                      throws IOException

Throws:

IOException

convertWALsToHFiles

protected void convertWALsToHFiles(Map<TableName,List<String>> tablesToWALFileList,
 Map<TableName,Long> tablesToPrevBackupTs)
                            throws IOException

Throws:

IOException

tableExists

protected boolean tableExists(TableName table,
 Connection conn)
                       throws IOException

Throws:

IOException

walToHFiles

protected void walToHFiles(List<String> dirPaths,
 List<String> tableList,
 long previousBackupTs)
                    throws IOException

Throws:

IOException

setBulkOutputPath

private void setBulkOutputPath(org.apache.hadoop.conf.Configuration conf,
 List<String> tableList)

incrementalCopyBulkloadHFiles

private void incrementalCopyBulkloadHFiles(org.apache.hadoop.fs.FileSystem tgtFs,
 TableName tn)
                                    throws IOException

Throws:

IOException

getBulkOutputDirForTable

protected org.apache.hadoop.fs.Path getBulkOutputDirForTable(TableName table)

Creates a path to the bulk load output directory for a table. This directory will look like: .../backupRoot/.tmp/backupId/namespace/table/data

Parameters:

table - The table whose HFiles are being bulk loaded

Returns:

A Path object representing the directory

getTmpBackupDirForTable

protected org.apache.hadoop.fs.Path getTmpBackupDirForTable(TableName table)

Creates a path to a table's directory within the temporary directory. This directory will look like: .../backupRoot/.tmp/backupId/namespace/table

Parameters:

table - The table whose HFiles are being bulk loaded

Returns:

A Path object representing the directory

getBulkOutputDir

protected org.apache.hadoop.fs.Path getBulkOutputDir()

Creates a path to a temporary backup directory. This directory will look like: .../backupRoot/.tmp/backupId

Returns:

A Path object representing the directory

getTmpBackupDir

protected org.apache.hadoop.fs.Path getTmpBackupDir()

Creates a path to the backup root directory's temporary subdirectory. The directory will look like: .../backupRoot/.tmp

Returns:

A Path object representing the directory

getTargetDirForTable

private org.apache.hadoop.fs.Path getTargetDirForTable(TableName table)

Creates a path to a destination directory for bulk loaded HFiles. This directory will look like: .../backupRoot/backupID/namespace/table

Parameters:

table - The table whose HFiles are being bulk loaded

Returns:

A Path object representing the directory

setupRegionLocator

private void setupRegionLocator()
                         throws IOException

Throws:

IOException

getFullBackupIds

private Map<TableName,String> getFullBackupIds()
                                        throws IOException

Throws:

IOException

verifyCfCompatibility

private void verifyCfCompatibility(Set<TableName> tables,
 Map<TableName,String> tablesToFullBackupId)
                            throws IOException,
ColumnFamilyMismatchException

Verifies that the current table descriptor CFs matches the descriptor CFs of the last full backup for the tables. This ensures CF compatibility across incremental backups. If a mismatch is detected, a full table backup should be taken, rather than an incremental one

Throws:

IOException

ColumnFamilyMismatchException

areCfsCompatible

private static boolean areCfsCompatible(ColumnFamilyDescriptor[] currentCfs,
 ColumnFamilyDescriptor[] backupCfs)