Field Detail

• NAME

  public static final String NAME

  See Also:

  Constant Field Values

• MAX_FILES_PER_REGION_PER_FAMILY

  public static final String MAX_FILES_PER_REGION_PER_FAMILY

  See Also:

  Constant Field Values

• CREATE_TABLE_CONF_KEY

  public static final String CREATE_TABLE_CONF_KEY

  See Also:

  Constant Field Values

Constructor Detail

• LoadIncrementalHFiles

  public LoadIncrementalHFiles(org.apache.hadoop.conf.Configuration conf)
                        throws Exception

  Throws:

  Exception

Method Detail

• setClusterIds

  public void setClusterIds(List<String> clusterIds)

• setDepth

  public void setDepth(int depth)

• doBulkLoad

  @Deprecated
  public void doBulkLoad(org.apache.hadoop.fs.Path hfofDir,
                           org.apache.hadoop.hbase.client.HTable table)
                  throws TableNotFoundException,
                         IOException

  Deprecated. since hbase-1.3.0, hbase-1.2.2, and hbase-1.1.6; replaced by doBulkLoad(Path, Admin, Table, RegionLocator).
  Perform a bulk load of the given directory into the given pre-existing table. This method is not threadsafe.

  Parameters:

  hfofDir - the directory that was provided as the output path of a job using HFileOutputFormat

  table - the table to load into

  Throws:

  TableNotFoundException - if table does not yet exist

  IOException

  See Also:

  HBASE-15876

• doBulkLoad

  public void doBulkLoad(org.apache.hadoop.fs.Path hfofDir,
                Admin admin,
                Table table,
                RegionLocator regionLocator)
                  throws TableNotFoundException,
                         IOException

  Perform a bulk load of the given directory into the given pre-existing table. This method is not threadsafe.

  Parameters:

  hfofDir - the directory that was provided as the output path of a job using HFileOutputFormat

  table - the table to load into

  Throws:

  TableNotFoundException - if table does not yet exist

  IOException

• prepareHFileQueue

  public void prepareHFileQueue(org.apache.hadoop.fs.Path hfilesDir,
                       Table table,
                       Deque<org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.LoadQueueItem> queue,
                       boolean validateHFile)
                         throws IOException

  Prepare a collection of LoadIncrementalHFiles.LoadQueueItem from list of source hfiles contained in the passed directory and validates whether the prepared queue has all the valid table column families in it.

  Parameters:

  hfilesDir - directory containing list of hfiles to be loaded into the table

  table - table to which hfiles should be loaded

  queue - queue which needs to be loaded into the table

  validateHFile - if true hfiles will be validated for its format

  Throws:

  IOException - If any I/O or network error occurred

• loadHFileQueue

  public void loadHFileQueue(Table table,
                    Connection conn,
                    Deque<org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.LoadQueueItem> queue,
                    Pair<byte[][],byte[][]> startEndKeys)
                      throws IOException

  Used by the replication sink to load the hfiles from the source cluster. It does the following, 1. groupOrSplitPhase(Table, ExecutorService, Deque, Pair) 2. bulkLoadPhase(Table, Connection, ExecutorService, Deque, Multimap)

  Parameters:

  table - Table to which these hfiles should be loaded to

  conn - Connection to use

  queue - LoadIncrementalHFiles.LoadQueueItem has hfiles yet to be loaded

  startEndKeys - starting and ending row keys of the region

  Throws:

  IOException

• bulkLoadPhase

  protected void bulkLoadPhase(Table table,
                   Connection conn,
                   ExecutorService pool,
                   Deque<org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.LoadQueueItem> queue,
                   com.google.common.collect.Multimap<ByteBuffer,org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.LoadQueueItem> regionGroups)
                        throws IOException

  This takes the LQI's grouped by likely regions and attempts to bulk load them. Any failures are re-queued for another pass with the groupOrSplitPhase.

  Throws:

  IOException

• splitStoreFile

  protected List<org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.LoadQueueItem> splitStoreFile(org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.LoadQueueItem item,
                                                                                           Table table,
                                                                                           byte[] startKey,
                                                                                           byte[] splitKey)
                                                                                                throws IOException

  Throws:

  IOException

• groupOrSplit

  protected List<org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.LoadQueueItem> groupOrSplit(com.google.common.collect.Multimap<ByteBuffer,org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.LoadQueueItem> regionGroups,
                                                                                         org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.LoadQueueItem item,
                                                                                         Table table,
                                                                                         Pair<byte[][],byte[][]> startEndKeys)
                                                                                              throws IOException

  Attempt to assign the given load queue item into its target region group. If the hfile boundary no longer fits into a region, physically splits the hfile such that the new bottom half will fit and returns the list of LQI's corresponding to the resultant hfiles. protected for testing

  Throws:

  IOException

• tryAtomicRegionLoad

  @Deprecated
  protected List<org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.LoadQueueItem> tryAtomicRegionLoad(HConnection conn,
                                                                                                           byte[] tableName,
                                                                                                           byte[] first,
                                                                                                           Collection<org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.LoadQueueItem> lqis)
                                                                                                     throws IOException

  Deprecated. As of release 0.96 (HBASE-9508). This will be removed in HBase 2.0.0. Use tryAtomicRegionLoad(Connection, TableName, byte[], Collection).

  Throws:

  IOException

• tryAtomicRegionLoad

  protected List<org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.LoadQueueItem> tryAtomicRegionLoad(Connection conn,
                                                                                                TableName tableName,
                                                                                                byte[] first,
                                                                                                Collection<org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.LoadQueueItem> lqis)
                                                                                                     throws IOException

  Attempts to do an atomic load of many hfiles into a region. If it fails, it returns a list of hfiles that need to be retried. If it is successful it will return an empty list. NOTE: To maintain row atomicity guarantees, region server callable should succeed atomically and fails atomically. Protected for testing.

  Returns:

  empty list if success, list of items to retry on recoverable failure

  Throws:

  IOException

• inferBoundaries

  public static byte[][] inferBoundaries(TreeMap<byte[],Integer> bdryMap)

• run

  public int run(String[] args)
          throws Exception

  Specified by:

  run in interface org.apache.hadoop.util.Tool

  Throws:

  Exception

• main

  public static void main(String[] args)
                   throws Exception

  Throws:

  Exception

• setBulkToken

  public void setBulkToken(String stagingDir)

  Called from replication sink, where it manages bulkToken(staging directory) by itself. This is used only when SecureBulkLoadEndpoint is configured in hbase.coprocessor.region.classes property. This directory is used as a temporary directory where all files are initially copied/moved from user given directory, set all the required file permissions and then from their it is finally loaded into a table. This should be set only when, one would like to manage the staging directory by itself. Otherwise this tool will handle this by itself.

  Parameters:

  stagingDir - staging directory path