Package org.apache.hadoop.hbase.backup.util

Class BackupUtils

java.lang.Object
org.apache.hadoop.hbase.backup.util.BackupUtils
@Private public final class BackupUtils extends Object
A collection for methods used by multiple classes to backup HBase tables.

  • Field Details

  • Constructor Details

  • Method Details

    • getRSLogTimestampMins

      public static Map<String,Long> getRSLogTimestampMins(Map<TableName,Map<String,Long>> rsLogTimestampMap)
      Loop through the RS log timestamp map for the tables, for each RS, find the min timestamp value for the RS among the tables.
      Parameters:
      rsLogTimestampMap - timestamp map
      Returns:
      the min timestamp of each RS

    • copyTableRegionInfo

      public static void copyTableRegionInfo(Connection conn, BackupInfo backupInfo, org.apache.hadoop.conf.Configuration conf) throws IOException
      copy out Table RegionInfo into incremental backup image need to consider move this logic into HBackupFileSystem
      Parameters:
      conn - connection
      backupInfo - backup info
      conf - configuration
      Throws:
      IOException - exception

    • writeRegioninfoOnFilesystem

      public static void writeRegioninfoOnFilesystem(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path regionInfoDir, RegionInfo regionInfo) throws IOException
      Write the .regioninfo file on-disk.
      Throws:
      IOException

    • parseHostNameFromLogFile

      public static String parseHostNameFromLogFile(org.apache.hadoop.fs.Path p)
      Parses hostname:port from WAL file path
      Parameters:
      p - path to WAL file
      Returns:
      hostname:port

    • getUniqueWALFileNamePart

      public static String getUniqueWALFileNamePart(String walFileName)
      Returns WAL file name
      Parameters:
      walFileName - WAL file name
      Returns:
      WAL file name

    • getUniqueWALFileNamePart

      public static String getUniqueWALFileNamePart(org.apache.hadoop.fs.Path p)
      Returns WAL file name
      Parameters:
      p - WAL file path
      Returns:
      WAL file name

    • getFilesLength

      public static long getFilesLength(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path dir) throws IOException
      Get the total length of files under the given directory recursively.
      Parameters:
      fs - The hadoop file system
      dir - The target directory
      Returns:
      the total length of files
      Throws:
      IOException - exception

    • getWALFilesOlderThan

      public static List<String> getWALFilesOlderThan(org.apache.hadoop.conf.Configuration c, HashMap<String,Long> hostTimestampMap) throws IOException
      Get list of all old WAL files (WALs and archive)
      Parameters:
      c - configuration
      hostTimestampMap - {host,timestamp} map
      Returns:
      list of WAL files
      Throws:
      IOException - exception

    • parseTableNames

      public static TableName[] parseTableNames(String tables)

    • checkPathExist

      public static boolean checkPathExist(String backupStr, org.apache.hadoop.conf.Configuration conf) throws IOException
      Check whether the backup path exist
      Parameters:
      backupStr - backup
      conf - configuration
      Returns:
      Yes if path exists
      Throws:
      IOException - exception

    • checkTargetDir

      public static void checkTargetDir(String backupRootPath, org.apache.hadoop.conf.Configuration conf) throws IOException
      Check target path first, confirm it doesn't exist before backup
      Parameters:
      backupRootPath - backup destination path
      conf - configuration
      Throws:
      IOException - exception

    • getMinValue

      public static <T> Long getMinValue(Map<T,Long> map)
      Get the min value for all the Values a map.
      Parameters:
      map - map
      Returns:
      the min value

    • parseHostFromOldLog

      public static String parseHostFromOldLog(org.apache.hadoop.fs.Path p)
      Parses host name:port from archived WAL path
      Parameters:
      p - path
      Returns:
      host name

    • getCreationTime

      public static Long getCreationTime(org.apache.hadoop.fs.Path p) throws IOException
      Given the log file, parse the timestamp from the file name. The timestamp is the last number.
      Parameters:
      p - a path to the log file
      Returns:
      the timestamp
      Throws:
      IOException - exception

    • getFiles

      public static List<String> getFiles(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path rootDir, List<String> files, org.apache.hadoop.fs.PathFilter filter) throws IOException
      Throws:
      IOException

    • cleanupBackupData

      public static void cleanupBackupData(BackupInfo context, org.apache.hadoop.conf.Configuration conf) throws IOException
      Throws:
      IOException

    • cleanupHLogDir

      private static void cleanupHLogDir(BackupInfo backupInfo, org.apache.hadoop.conf.Configuration conf) throws IOException
      Clean up directories which are generated when DistCp copying hlogs
      Parameters:
      backupInfo - backup info
      conf - configuration
      Throws:
      IOException - exception

    • cleanupTargetDir

      private static void cleanupTargetDir(BackupInfo backupInfo, org.apache.hadoop.conf.Configuration conf)

    • getTableBackupDir

      public static String getTableBackupDir(String backupRootDir, String backupId, TableName tableName)
      Given the backup root dir, backup id and the table name, return the backup image location, which is also where the backup manifest file is. return value look like: "hdfs://backup.hbase.org:9000/user/biadmin/backup1/backup_1396650096738/default/t1_dn/"
      Parameters:
      backupRootDir - backup root directory
      backupId - backup id
      tableName - table name
      Returns:
      backupPath String for the particular table

    • sortHistoryListDesc

      public static ArrayList<BackupInfo> sortHistoryListDesc(ArrayList<BackupInfo> historyList)
      Sort history list by start time in descending order.
      Parameters:
      historyList - history list
      Returns:
      sorted list of BackupCompleteData

    • listStatus

      public static org.apache.hadoop.fs.FileStatus[] listStatus(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path dir, org.apache.hadoop.fs.PathFilter filter) throws IOException
      Calls fs.listStatus() and treats FileNotFoundException as non-fatal This accommodates differences between hadoop versions, where hadoop 1 does not throw a FileNotFoundException, and return an empty FileStatus[] while Hadoop 2 will throw FileNotFoundException.
      Parameters:
      fs - file system
      dir - directory
      filter - path filter
      Returns:
      null if dir is empty or doesn't exist, otherwise FileStatus array
      Throws:
      IOException

    • getPath

      public static String getPath(org.apache.hadoop.fs.Path p)
      Return the 'path' component of a Path. In Hadoop, Path is a URI. This method returns the 'path' component of a Path's URI: e.g. If a Path is hdfs://example.org:9000/hbase_trunk/TestTable/compaction.dir, this method returns /hbase_trunk/TestTable/compaction.dir. This method is useful if you want to print out a Path without qualifying Filesystem instance.
      Parameters:
      p - file system Path whose 'path' component we are to return.
      Returns:
      Path portion of the Filesystem

    • getLogBackupDir

      public static String getLogBackupDir(String backupRootDir, String backupId)
      Given the backup root dir and the backup id, return the log file location for an incremental backup.
      Parameters:
      backupRootDir - backup root directory
      backupId - backup id
      Returns:
      logBackupDir: ".../user/biadmin/backup1/WALs/backup_1396650096738"

    • getHistory

      private static List<BackupInfo> getHistory(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path backupRootPath) throws IOException
      Throws:
      IOException

    • getHistory

      public static List<BackupInfo> getHistory(org.apache.hadoop.conf.Configuration conf, int n, org.apache.hadoop.fs.Path backupRootPath, BackupInfo.Filter... filters) throws IOException
      Throws:
      IOException

    • loadBackupInfo

      public static BackupInfo loadBackupInfo(org.apache.hadoop.fs.Path backupRootPath, String backupId, org.apache.hadoop.fs.FileSystem fs) throws IOException
      Throws:
      IOException

    • createRestoreRequest

      public static RestoreRequest createRestoreRequest(String backupRootDir, String backupId, boolean check, TableName[] fromTables, TableName[] toTables, boolean isOverwrite)
      Create restore request.
      Parameters:
      backupRootDir - backup root dir
      backupId - backup id
      check - check only
      fromTables - table list from
      toTables - table list to
      isOverwrite - overwrite data
      Returns:
      request obkect

    • validate

      public static boolean validate(HashMap<TableName,BackupManifest> backupManifestMap, org.apache.hadoop.conf.Configuration conf) throws IOException
      Throws:
      IOException

    • getBulkOutputDir

      public static org.apache.hadoop.fs.Path getBulkOutputDir(org.apache.hadoop.fs.Path restoreRootDir, String tableName, org.apache.hadoop.conf.Configuration conf, boolean deleteOnExit) throws IOException
      Throws:
      IOException

    • getBulkOutputDir

      public static org.apache.hadoop.fs.Path getBulkOutputDir(org.apache.hadoop.fs.Path restoreRootDir, String tableName, org.apache.hadoop.conf.Configuration conf) throws IOException
      Throws:
      IOException

    • getBulkOutputDir

      public static org.apache.hadoop.fs.Path getBulkOutputDir(String tableName, org.apache.hadoop.conf.Configuration conf, boolean deleteOnExit) throws IOException
      Throws:
      IOException

    • getTmpRestoreOutputDir

      public static org.apache.hadoop.fs.Path getTmpRestoreOutputDir(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.conf.Configuration conf)
      Build temporary output path
      Parameters:
      fs - filesystem for default output dir
      conf - configuration
      Returns:
      output path

    • getFileNameCompatibleString

      public static String getFileNameCompatibleString(TableName table)

    • failed

      public static boolean failed(int result)

    • succeeded

      public static boolean succeeded(int result)

    • createLoader

      public static BulkLoadHFiles createLoader(org.apache.hadoop.conf.Configuration config)

    • findMostRecentBackupId

      public static String findMostRecentBackupId(String[] backupIds)