Package org.apache.hadoop.hbase.backup.util

Class BackupUtils

java.lang.Object
org.apache.hadoop.hbase.backup.util.BackupUtils
@Private public final class BackupUtils extends Object
A collection for methods used by multiple classes to backup HBase tables.

  • Field Details

  • Constructor Details

  • Method Details

    • getRSLogTimestampMins

      public static Map<String,Long> getRSLogTimestampMins(Map<TableName,Map<String,Long>> rsLogTimestampMap)
      Loop through the RS log timestamp map for the tables, for each RS, find the min timestamp value for the RS among the tables.
      Parameters:
      rsLogTimestampMap - timestamp map
      Returns:
      the min timestamp of each RS

    • copyTableRegionInfo

      public static void copyTableRegionInfo(Connection conn, BackupInfo backupInfo, org.apache.hadoop.conf.Configuration conf) throws IOException
      copy out Table RegionInfo into incremental backup image need to consider move this logic into HBackupFileSystem
      Parameters:
      conn - connection
      backupInfo - backup info
      conf - configuration
      Throws:
      IOException - exception

    • writeRegioninfoOnFilesystem

      public static void writeRegioninfoOnFilesystem(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path regionInfoDir, RegionInfo regionInfo) throws IOException
      Write the .regioninfo file on-disk.
      Throws:
      IOException

    • parseHostNameFromLogFile

      public static String parseHostNameFromLogFile(org.apache.hadoop.fs.Path p)
      Parses hostname:port from WAL file path
      Parameters:
      p - path to WAL file
      Returns:
      hostname:port

    • getUniqueWALFileNamePart

      public static String getUniqueWALFileNamePart(String walFileName)
      Returns WAL file name
      Parameters:
      walFileName - WAL file name
      Returns:
      WAL file name

    • getUniqueWALFileNamePart

      public static String getUniqueWALFileNamePart(org.apache.hadoop.fs.Path p)
      Returns WAL file name
      Parameters:
      p - WAL file path
      Returns:
      WAL file name

    • getFilesLength

      public static long getFilesLength(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path dir) throws IOException
      Get the total length of files under the given directory recursively.
      Parameters:
      fs - The hadoop file system
      dir - The target directory
      Returns:
      the total length of files
      Throws:
      IOException - exception

    • getWALFilesOlderThan

      public static List<String> getWALFilesOlderThan(org.apache.hadoop.conf.Configuration c, HashMap<String,Long> hostTimestampMap) throws IOException
      Get list of all old WAL files (WALs and archive)
      Parameters:
      c - configuration
      hostTimestampMap - {host,timestamp} map
      Returns:
      list of WAL files
      Throws:
      IOException - exception

    • parseTableNames

      public static TableName[] parseTableNames(String tables)

    • checkPathExist

      public static boolean checkPathExist(String backupStr, org.apache.hadoop.conf.Configuration conf) throws IOException
      Check whether the backup path exist
      Parameters:
      backupStr - backup
      conf - configuration
      Returns:
      Yes if path exists
      Throws:
      IOException - exception

    • checkTargetDir

      public static void checkTargetDir(String backupRootPath, org.apache.hadoop.conf.Configuration conf) throws IOException
      Check target path first, confirm it doesn't exist before backup
      Parameters:
      backupRootPath - backup destination path
      conf - configuration
      Throws:
      IOException - exception

    • getMinValue

      public static <T> Long getMinValue(Map<T,Long> map)
      Get the min value for all the Values a map.
      Parameters:
      map - map
      Returns:
      the min value

    • parseHostFromOldLog

      public static String parseHostFromOldLog(org.apache.hadoop.fs.Path p)
      Parses host name:port from archived WAL path
      Parameters:
      p - path
      Returns:
      host name

    • getCreationTime

      public static Long getCreationTime(org.apache.hadoop.fs.Path p) throws IOException
      Given the log file, parse the timestamp from the file name. The timestamp is the last number.
      Parameters:
      p - a path to the log file
      Returns:
      the timestamp
      Throws:
      IOException - exception

    • getFiles

      public static List<String> getFiles(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path rootDir, List<String> files, org.apache.hadoop.fs.PathFilter filter) throws IOException
      Throws:
      IOException

    • cleanupBackupData

      public static void cleanupBackupData(BackupInfo context, org.apache.hadoop.conf.Configuration conf) throws IOException
      Throws:
      IOException

    • cleanupHLogDir

      private static void cleanupHLogDir(BackupInfo backupInfo, org.apache.hadoop.conf.Configuration conf) throws IOException
      Clean up directories which are generated when DistCp copying hlogs
      Parameters:
      backupInfo - backup info
      conf - configuration
      Throws:
      IOException - exception

    • cleanupTargetDir

      private static void cleanupTargetDir(BackupInfo backupInfo, org.apache.hadoop.conf.Configuration conf)

    • getTableBackupDir

      public static String getTableBackupDir(String backupRootDir, String backupId, TableName tableName)
      Given the backup root dir, backup id and the table name, return the backup image location, which is also where the backup manifest file is. return value look like: "hdfs://backup.hbase.org:9000/user/biadmin/backup1/backup_1396650096738/default/t1_dn/"
      Parameters:
      backupRootDir - backup root directory
      backupId - backup id
      tableName - table name
      Returns:
      backupPath String for the particular table

    • listStatus

      public static org.apache.hadoop.fs.FileStatus[] listStatus(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path dir, org.apache.hadoop.fs.PathFilter filter) throws IOException
      Calls fs.listStatus() and treats FileNotFoundException as non-fatal This accommodates differences between hadoop versions, where hadoop 1 does not throw a FileNotFoundException, and return an empty FileStatus[] while Hadoop 2 will throw FileNotFoundException.
      Parameters:
      fs - file system
      dir - directory
      filter - path filter
      Returns:
      null if dir is empty or doesn't exist, otherwise FileStatus array
      Throws:
      IOException

    • getPath

      public static String getPath(org.apache.hadoop.fs.Path p)
      Return the 'path' component of a Path. In Hadoop, Path is a URI. This method returns the 'path' component of a Path's URI: e.g. If a Path is hdfs://example.org:9000/hbase_trunk/TestTable/compaction.dir, this method returns /hbase_trunk/TestTable/compaction.dir. This method is useful if you want to print out a Path without qualifying Filesystem instance.
      Parameters:
      p - file system Path whose 'path' component we are to return.
      Returns:
      Path portion of the Filesystem

    • getLogBackupDir

      public static String getLogBackupDir(String backupRootDir, String backupId)
      Given the backup root dir and the backup id, return the log file location for an incremental backup.
      Parameters:
      backupRootDir - backup root directory
      backupId - backup id
      Returns:
      logBackupDir: ".../user/biadmin/backup1/WALs/backup_1396650096738"

    • getHistory

      private static List<BackupInfo> getHistory(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path backupRootPath) throws IOException
      Loads all backup history as stored in files on the given backup root path.
      Returns:
      all backup history, from newest (most recent) to oldest (least recent)
      Throws:
      IOException

    • getHistory

      public static List<BackupInfo> getHistory(org.apache.hadoop.conf.Configuration conf, int n, org.apache.hadoop.fs.Path backupRootPath, BackupInfo.Filter... filters) throws IOException
      Loads all backup history as stored in files on the given backup root path, and returns the first n entries matching all given filters.
      Returns:
      (subset of) backup history, from newest (most recent) to oldest (least recent)
      Throws:
      IOException

    • loadBackupInfo

      public static BackupInfo loadBackupInfo(org.apache.hadoop.fs.Path backupRootPath, String backupId, org.apache.hadoop.fs.FileSystem fs) throws IOException
      Throws:
      IOException

    • createRestoreRequest

      public static RestoreRequest createRestoreRequest(String backupRootDir, String backupId, boolean check, TableName[] fromTables, TableName[] toTables, boolean isOverwrite)
      Create restore request.
      Parameters:
      backupRootDir - backup root dir
      backupId - backup id
      check - check only
      fromTables - table list from
      toTables - table list to
      isOverwrite - overwrite data
      Returns:
      request obkect

    • createRestoreRequest

      public static RestoreRequest createRestoreRequest(String backupRootDir, String backupId, boolean check, TableName[] fromTables, TableName[] toTables, boolean isOverwrite, boolean isKeepOriginalSplits)

    • validate

      public static boolean validate(List<TableName> tables, BackupManifest backupManifest, org.apache.hadoop.conf.Configuration conf) throws IOException
      Throws:
      IOException

    • getBulkOutputDir

      public static org.apache.hadoop.fs.Path getBulkOutputDir(org.apache.hadoop.fs.Path restoreRootDir, String tableName, org.apache.hadoop.conf.Configuration conf, boolean deleteOnExit) throws IOException
      Throws:
      IOException

    • getBulkOutputDir

      public static org.apache.hadoop.fs.Path getBulkOutputDir(org.apache.hadoop.fs.Path restoreRootDir, String tableName, org.apache.hadoop.conf.Configuration conf) throws IOException
      Throws:
      IOException

    • getBulkOutputDir

      public static org.apache.hadoop.fs.Path getBulkOutputDir(String tableName, org.apache.hadoop.conf.Configuration conf, boolean deleteOnExit) throws IOException
      Throws:
      IOException

    • getTmpRestoreOutputDir

      public static org.apache.hadoop.fs.Path getTmpRestoreOutputDir(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.conf.Configuration conf)
      Build temporary output path
      Parameters:
      fs - filesystem for default output dir
      conf - configuration
      Returns:
      output path

    • getFileNameCompatibleString

      public static String getFileNameCompatibleString(TableName table)

    • failed

      public static boolean failed(int result)

    • succeeded

      public static boolean succeeded(int result)

    • createLoader

      public static BulkLoadHFiles createLoader(org.apache.hadoop.conf.Configuration config)

    • findMostRecentBackupId

      public static String findMostRecentBackupId(String[] backupIds)

    • logRoll

      public static void logRoll(Connection conn, String backupRootDir, org.apache.hadoop.conf.Configuration conf) throws IOException
      roll WAL writer for all region servers and record the newest log roll result
      Throws:
      IOException

    • logRollV1

      private static void logRollV1(Connection conn, String backupRootDir) throws IOException
      Throws:
      IOException

    • logRollV2

      private static void logRollV2(Connection conn, String backupRootDir) throws IOException
      Throws:
      IOException

    • getReplicationCheckpoint

      public static long getReplicationCheckpoint(Connection conn) throws IOException
      Calculates the replication checkpoint timestamp used for continuous backup.

      A replication checkpoint is the earliest timestamp across all region servers such that every WAL entry before that point is known to be replicated to the target system. This is essential for features like Point-in-Time Restore (PITR) and incremental backups, where we want to confidently restore data to a consistent state without missing updates.

      The checkpoint is calculated using a combination of:

      • The start timestamps of WAL files currently being replicated for each server.
      • The latest successfully replicated timestamp recorded by the replication marker chore.

      We combine these two sources to handle the following challenges:

      • Stale WAL start times: If replication traffic is low or WALs are long-lived, the replication offset may point to the same WAL for a long time, resulting in stale timestamps that underestimate progress. This could delay PITR unnecessarily.
      • Limitations of marker-only tracking: The replication marker chore stores the last successfully replicated timestamp per region server in a system table. However, this data may become stale if the server goes offline or region ownership changes. For example, if a region initially belonged to rs1 and was later moved to rs4 due to re-balancing, rs1’s marker would persist even though it no longer holds any regions. Relying solely on these stale markers could lead to incorrect or outdated checkpoints.

      To handle these limitations, the method:

      1. Verifies that the continuous backup peer exists to ensure replication is enabled.
      2. Retrieves WAL replication queue information for the peer, collecting WAL start times per region server. This gives us a lower bound for replication progress.
      3. Reads the marker chore's replicated timestamps from the backup system table.
      4. For servers found in both sources, if the marker timestamp is more recent than the WAL's start timestamp, we use the marker (since replication has progressed beyond the WAL).
      5. We discard marker entries for region servers that are not present in WAL queues, assuming those servers are no longer relevant (e.g., decommissioned or reassigned).
      6. The checkpoint is the minimum of all chosen timestamps — i.e., the slowest replicating region server.
      7. Finally, we persist the updated marker information to include any newly participating region servers.

      Note: If the replication marker chore is disabled, we fall back to using only the WAL start times. This ensures correctness but may lead to conservative checkpoint estimates during idle periods.

      Parameters:
      conn - the HBase connection
      Returns:
      the calculated replication checkpoint timestamp
      Throws:
      IOException - if reading replication queues or updating the backup system table fails

    • continuousBackupReplicationPeerExists

      private static boolean continuousBackupReplicationPeerExists(Admin admin) throws IOException
      Throws:
      IOException

    • formatToDateString

      public static String formatToDateString(long dayInMillis)
      Convert dayInMillis to "yyyy-MM-dd" format

    • collectBulkFiles

      public static List<org.apache.hadoop.fs.Path> collectBulkFiles(Connection conn, TableName sourceTable, TableName targetTable, long startTime, long endTime, org.apache.hadoop.fs.Path restoreRootDir, List<String> walDirs) throws IOException
      Fetches bulkload filepaths based on the given time range from backup WAL directory.
      Throws:
      IOException

    • getValidWalDirs

      public static List<String> getValidWalDirs(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path walBackupDir, long startTime, long endTime) throws IOException
      Fetches valid WAL directories based on the given time range.
      Throws:
      IOException