Package org.apache.hadoop.hbase.backup.mapreduce

Class BulkLoadCollectorJob

java.lang.Object
org.apache.hadoop.conf.Configured
org.apache.hadoop.hbase.backup.mapreduce.BulkLoadCollectorJob
All Implemented Interfaces:
org.apache.hadoop.conf.Configurable, org.apache.hadoop.util.Tool
@Private public class BulkLoadCollectorJob extends org.apache.hadoop.conf.Configured implements org.apache.hadoop.util.Tool
MapReduce job that scans WAL backups and extracts referenced bulk-load store-file paths.

This job is intended to be used when you want a list of HFiles / store-files referenced by WAL bulk-load descriptors. It emits a de-duplicated list of full paths (one per line) by default using the BulkLoadCollectorJob.DedupReducer.

Usage (CLI): BulkLoadCollector <WAL inputdir> <bulk-files-output-dir> [<tables> [<tableMappings>]]

  • Nested Class Summary

    Nested Classes
    Modifier and Type
    Class
    Description
    static class 
    BulkLoadCollectorJob.BulkLoadCollectorMapper
    Mapper that extracts relative bulk-load paths from a WAL entry (via BulkLoadProcessor), resolves them to full paths (via BackupFileSystemManager#resolveBulkLoadFullPath(Path, Path)), and emits each full path as the map key (Text).
    static class 
    BulkLoadCollectorJob.DedupReducer
    Reducer that deduplicates full-path keys emitted by the mappers.

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final String
    DEFAULT_REDUCERS
     
    private static final org.slf4j.Logger
    LOG
     
    static final String
    NAME
     

  • Constructor Summary

    Constructors
    Constructor
    Description
    BulkLoadCollectorJob()
     
    BulkLoadCollectorJob(org.apache.hadoop.conf.Configuration c)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    org.apache.hadoop.mapreduce.Job
    createSubmittableJob(String[] args)
    Create and configure a Job instance for bulk-file collection.
    private static org.apache.hadoop.mapreduce.Job
    createSubmittableJob(org.apache.hadoop.conf.Configuration conf, String inputDirs, String bulkFilesOut)
    Low-level job wiring.
    static void
    main(String[] args)
    CLI entry point.
    int
    run(String[] args)
     
    private void
    setupTime(org.apache.hadoop.conf.Configuration conf, String option)
    Parse a time option.
    private void
    usage(String errorMsg)
    Print usage/help for the BulkLoadCollectorJob CLI/driver.

    Methods inherited from class org.apache.hadoop.conf.Configured

    getConf, setConf

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    Methods inherited from interface org.apache.hadoop.conf.Configurable

    getConf, setConf

  • Field Details

  • Constructor Details

  • Method Details

    • createSubmittableJob

      public org.apache.hadoop.mapreduce.Job createSubmittableJob(String[] args) throws IOException
      Create and configure a Job instance for bulk-file collection.
      Parameters:
      args - CLI args expected to be: inputDirs bulkFilesOut [tables] [tableMap]
      Throws:
      IOException - on misconfiguration

    • createSubmittableJob

      private static org.apache.hadoop.mapreduce.Job createSubmittableJob(org.apache.hadoop.conf.Configuration conf, String inputDirs, String bulkFilesOut) throws IOException
      Low-level job wiring. Creates the Job instance and sets input, mapper, reducer and output.
      Parameters:
      conf - configuration used for the job
      inputDirs - WAL input directories (comma-separated)
      bulkFilesOut - output directory to write discovered full-paths
      Throws:
      IOException - on invalid args

    • setupTime

      private void setupTime(org.apache.hadoop.conf.Configuration conf, String option) throws IOException
      Parse a time option. Supports the user-friendly ISO-like format yyyy-MM-dd'T'HH:mm:ss.SS or milliseconds since epoch. If the option is not present, this method is a no-op.
      Parameters:
      conf - configuration containing option
      option - key to read (e.g. WALInputFormat.START_TIME_KEY)
      Throws:
      IOException - on parse failure

    • main

      public static void main(String[] args) throws Exception
      CLI entry point.
      Parameters:
      args - job arguments (see usage(String))
      Throws:
      Exception - on job failure

    • run

      public int run(String[] args) throws Exception
      Specified by:
      run in interface org.apache.hadoop.util.Tool
      Throws:
      Exception

    • usage

      private void usage(String errorMsg)
      Print usage/help for the BulkLoadCollectorJob CLI/driver. 

       args layout:
   args[0] = input directory (required)
   args[1] = output directory (required)
   args[2] = tables (comma-separated) (optional)
   args[3] = tableMappings (comma-separated) (optional; must match tables length)