org.apache.hadoop.hbase.tool

Class LoadIncrementalHFiles

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.hbase.tool.LoadIncrementalHFiles

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, org.apache.hadoop.util.Tool

Direct Known Subclasses:

BulkLoadHFilesTool, LoadIncrementalHFiles

Deprecated.

since 2.2.0, will be removed in 3.0.0. Use BulkLoadHFiles instead. Please rewrite your code if you rely on methods other than the run(Map, TableName) and run(String, TableName), as all the methods other than them will be removed with no replacement.

@Deprecated
 @InterfaceAudience.Public
public class LoadIncrementalHFiles
extends org.apache.hadoop.conf.Configured
implements org.apache.hadoop.util.Tool

Tool to load the output of HFileOutputFormat into an existing table.

Notice that, by default, this class should be kept till 4.0.0, but as this is a bad practice that we expose an implementation class instead of an interface, we want to fix it ASAP. That's why we will remove this class completely in 3.0.0. Please change your code to use BulkLoadHFiles.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
private static interface	LoadIncrementalHFiles.BulkHFileVisitor<TFamily> Deprecated.
static class	LoadIncrementalHFiles.LoadQueueItem Deprecated. since 2.2.0 and will be removed in 3.0.0. Use BulkLoadHFiles instead.

Field Summary

Fields

Modifier and Type	Field and Description
static String	ALWAYS_COPY_FILES Deprecated.
private static String	ASSIGN_SEQ_IDS Deprecated.
private boolean	assignSeqIds Deprecated.
private boolean	bulkLoadByFamily Deprecated.
private String	bulkToken Deprecated.
private List<String>	clusterIds Deprecated.
static String	CREATE_TABLE_CONF_KEY Deprecated.
static String	FAIL_IF_NEED_SPLIT_HFILE Deprecated.
private boolean	failIfNeedSplitHFile Deprecated.
private FsDelegationToken	fsDelegationToken Deprecated.
static String	IGNORE_UNMATCHED_CF_CONF_KEY Deprecated.
private static org.slf4j.Logger	LOG Deprecated.
static String	MAX_FILES_PER_REGION_PER_FAMILY Deprecated.
private int	maxFilesPerRegionPerFamily Deprecated.
static String	NAME Deprecated. since 2.2.0, will be removed in 3.0.0, with no replacement. End user should not depend on this value.
private int	nrThreads Deprecated.
private AtomicInteger	numRetries Deprecated.
private boolean	replicate Deprecated.
(package private) static String	RETRY_ON_IO_EXCEPTION Deprecated.
private RpcControllerFactory	rpcControllerFactory Deprecated.
(package private) static String	TMP_DIR Deprecated.
private UserProvider	userProvider Deprecated.

Constructor Summary

Constructors

Constructor and Description
LoadIncrementalHFiles(org.apache.hadoop.conf.Configuration conf) Deprecated.

Method Summary

Modifier and Type	Method and Description
protected ClientServiceCallable<byte[]>	buildClientServiceCallable(Connection conn, TableName tableName, byte[] first, Collection<LoadIncrementalHFiles.LoadQueueItem> lqis, boolean copyFile) Deprecated.
protected void	bulkLoadPhase(Table table, Connection conn, ExecutorService pool, Deque<LoadIncrementalHFiles.LoadQueueItem> queue, org.apache.hbase.thirdparty.com.google.common.collect.Multimap<ByteBuffer,LoadIncrementalHFiles.LoadQueueItem> regionGroups, boolean copyFile, Map<LoadIncrementalHFiles.LoadQueueItem,ByteBuffer> item2RegionMap) Deprecated. This takes the LQI's grouped by likely regions and attempts to bulk load them.
private boolean	checkHFilesCountPerRegionPerFamily(org.apache.hbase.thirdparty.com.google.common.collect.Multimap<ByteBuffer,LoadIncrementalHFiles.LoadQueueItem> regionGroups) Deprecated.
private void	checkRegionIndexValid(int idx, Pair<byte[][],byte[][]> startEndKeys, TableName tableName) Deprecated. we can consider there is a region hole in following conditions.
private void	cleanup(Admin admin, Deque<LoadIncrementalHFiles.LoadQueueItem> queue, ExecutorService pool, SecureBulkLoadClient secureClient) Deprecated.
private static void	copyHFileHalf(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path inFile, org.apache.hadoop.fs.Path outFile, Reference reference, ColumnFamilyDescriptor familyDescriptor) Deprecated. Copy half of an HFile into a new HFile.
private ExecutorService	createExecutorService() Deprecated.
private void	createTable(TableName tableName, org.apache.hadoop.fs.Path hfofDir, Admin admin) Deprecated. If the table is created for the first time, then "completebulkload" reads the files twice.
void	disableReplication() Deprecated. Disables replication for these bulkloaded files.
private void	discoverLoadQueue(Deque<LoadIncrementalHFiles.LoadQueueItem> ret, org.apache.hadoop.fs.Path hfofDir, boolean validateHFile) Deprecated. Walk the given directory for all HFiles, and return a Queue containing all such files.
Map<LoadIncrementalHFiles.LoadQueueItem,ByteBuffer>	doBulkLoad(Map<byte[],List<org.apache.hadoop.fs.Path>> map, Admin admin, Table table, RegionLocator regionLocator, boolean silence, boolean copyFile) Deprecated. Perform a bulk load of the given directory into the given pre-existing table.
Map<LoadIncrementalHFiles.LoadQueueItem,ByteBuffer>	doBulkLoad(org.apache.hadoop.fs.Path hfofDir, Admin admin, Table table, RegionLocator regionLocator) Deprecated. Perform a bulk load of the given directory into the given pre-existing table.
Map<LoadIncrementalHFiles.LoadQueueItem,ByteBuffer>	doBulkLoad(org.apache.hadoop.fs.Path hfofDir, Admin admin, Table table, RegionLocator regionLocator, boolean silence, boolean copyFile) Deprecated. Perform a bulk load of the given directory into the given pre-existing table.
private int	getRegionIndex(Pair<byte[][],byte[][]> startEndKeys, byte[] key) Deprecated.
private String	getUniqueName() Deprecated.
private Map<byte[],Collection<LoadIncrementalHFiles.LoadQueueItem>>	groupByFamilies(Collection<LoadIncrementalHFiles.LoadQueueItem> itemsInRegion) Deprecated.
protected Pair<List<LoadIncrementalHFiles.LoadQueueItem>,String>	groupOrSplit(org.apache.hbase.thirdparty.com.google.common.collect.Multimap<ByteBuffer,LoadIncrementalHFiles.LoadQueueItem> regionGroups, LoadIncrementalHFiles.LoadQueueItem item, Table table, Pair<byte[][],byte[][]> startEndKeys) Deprecated. Attempt to assign the given load queue item into its target region group.
private Pair<org.apache.hbase.thirdparty.com.google.common.collect.Multimap<ByteBuffer,LoadIncrementalHFiles.LoadQueueItem>,Set<String>>	groupOrSplitPhase(Table table, ExecutorService pool, Deque<LoadIncrementalHFiles.LoadQueueItem> queue, Pair<byte[][],byte[][]> startEndKeys) Deprecated.
static byte[][]	inferBoundaries(SortedMap<byte[],Integer> bdryMap) Deprecated. Infers region boundaries for a new table.
void	initialize() Deprecated.
private boolean	isAlwaysCopyFiles() Deprecated.
private boolean	isCreateTable() Deprecated.
private boolean	isSecureBulkLoadEndpointAvailable() Deprecated.
private boolean	isSilence() Deprecated.
void	loadHFileQueue(Table table, Connection conn, Deque<LoadIncrementalHFiles.LoadQueueItem> queue, Pair<byte[][],byte[][]> startEndKeys) Deprecated. Used by the replication sink to load the hfiles from the source cluster.
void	loadHFileQueue(Table table, Connection conn, Deque<LoadIncrementalHFiles.LoadQueueItem> queue, Pair<byte[][],byte[][]> startEndKeys, boolean copyFile) Deprecated. Used by the replication sink to load the hfiles from the source cluster.
static void	main(String[] args) Deprecated.
private Map<LoadIncrementalHFiles.LoadQueueItem,ByteBuffer>	performBulkLoad(Admin admin, Table table, RegionLocator regionLocator, Deque<LoadIncrementalHFiles.LoadQueueItem> queue, ExecutorService pool, SecureBulkLoadClient secureClient, boolean copyFile) Deprecated.
private void	populateLoadQueue(Deque<LoadIncrementalHFiles.LoadQueueItem> ret, Map<byte[],List<org.apache.hadoop.fs.Path>> map) Deprecated. Populate the Queue with given HFiles
void	prepareHFileQueue(Map<byte[],List<org.apache.hadoop.fs.Path>> map, Table table, Deque<LoadIncrementalHFiles.LoadQueueItem> queue, boolean silence) Deprecated. Prepare a collection of LoadIncrementalHFiles.LoadQueueItem from list of source hfiles contained in the passed directory and validates whether the prepared queue has all the valid table column families in it.
void	prepareHFileQueue(org.apache.hadoop.fs.Path hfilesDir, Table table, Deque<LoadIncrementalHFiles.LoadQueueItem> queue, boolean validateHFile) Deprecated. Prepare a collection of LoadIncrementalHFiles.LoadQueueItem from list of source hfiles contained in the passed directory and validates whether the prepared queue has all the valid table column families in it.
void	prepareHFileQueue(org.apache.hadoop.fs.Path hfilesDir, Table table, Deque<LoadIncrementalHFiles.LoadQueueItem> queue, boolean validateHFile, boolean silence) Deprecated. Prepare a collection of LoadIncrementalHFiles.LoadQueueItem from list of source hfiles contained in the passed directory and validates whether the prepared queue has all the valid table column families in it.
Map<LoadIncrementalHFiles.LoadQueueItem,ByteBuffer>	run(Map<byte[],List<org.apache.hadoop.fs.Path>> family2Files, TableName tableName) Deprecated. Perform bulk load on the given table.
protected Map<LoadIncrementalHFiles.LoadQueueItem,ByteBuffer>	run(org.apache.hadoop.fs.Path hfofDir, TableName tableName) Deprecated.
int	run(String[] args) Deprecated.
Map<LoadIncrementalHFiles.LoadQueueItem,ByteBuffer>	run(String hfofDir, TableName tableName) Deprecated. Perform bulk load on the given table.
void	setBulkToken(String stagingDir) Deprecated. Called from replication sink, where it manages bulkToken(staging directory) by itself.
void	setClusterIds(List<String> clusterIds) Deprecated.
private static boolean	shouldCopyHFileMetaKey(byte[] key) Deprecated.
(package private) static void	splitStoreFile(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path inFile, ColumnFamilyDescriptor familyDesc, byte[] splitKey, org.apache.hadoop.fs.Path bottomOut, org.apache.hadoop.fs.Path topOut) Deprecated. Split a storefile into a top and bottom half, maintaining the metadata, recreating bloom filters, etc.
private List<LoadIncrementalHFiles.LoadQueueItem>	splitStoreFile(LoadIncrementalHFiles.LoadQueueItem item, Table table, byte[] startKey, byte[] splitKey) Deprecated.
private String	toString(List<Pair<byte[],String>> list) Deprecated.
protected List<LoadIncrementalHFiles.LoadQueueItem>	tryAtomicRegionLoad(ClientServiceCallable<byte[]> serviceCallable, TableName tableName, byte[] first, Collection<LoadIncrementalHFiles.LoadQueueItem> lqis) Deprecated. as of release 2.3.0. Use BulkLoadHFiles instead.
protected List<LoadIncrementalHFiles.LoadQueueItem>	tryAtomicRegionLoad(Connection conn, TableName tableName, byte[] first, Collection<LoadIncrementalHFiles.LoadQueueItem> lqis, boolean copyFile) Deprecated. as of release 2.3.0. Use BulkLoadHFiles instead.
private void	usage() Deprecated.
private void	validateFamiliesInHFiles(Table table, Deque<LoadIncrementalHFiles.LoadQueueItem> queue, boolean silence) Deprecated. Checks whether there is any invalid family name in HFiles to be bulk loaded.
private static <TFamily> void	visitBulkHFiles(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path bulkDir, LoadIncrementalHFiles.BulkHFileVisitor<TFamily> visitor) Deprecated. Iterate over the bulkDir hfiles.
private static <TFamily> void	visitBulkHFiles(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path bulkDir, LoadIncrementalHFiles.BulkHFileVisitor<TFamily> visitor, boolean validateHFile) Deprecated. Iterate over the bulkDir hfiles.

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.conf.Configurable

getConf, setConf

Field Detail

LOG

private static final org.slf4j.Logger LOG

Deprecated.

NAME

@Deprecated
public static final String NAME

Deprecated. since 2.2.0, will be removed in 3.0.0, with no replacement. End user should not depend on this value.

See Also:

Constant Field Values

RETRY_ON_IO_EXCEPTION

static final String RETRY_ON_IO_EXCEPTION

Deprecated.

See Also:

Constant Field Values

MAX_FILES_PER_REGION_PER_FAMILY

public static final String MAX_FILES_PER_REGION_PER_FAMILY

Deprecated.

See Also:

Constant Field Values

ASSIGN_SEQ_IDS

private static final String ASSIGN_SEQ_IDS

Deprecated.

See Also:

Constant Field Values

CREATE_TABLE_CONF_KEY

public static final String CREATE_TABLE_CONF_KEY

Deprecated.

See Also:

Constant Field Values

IGNORE_UNMATCHED_CF_CONF_KEY

public static final String IGNORE_UNMATCHED_CF_CONF_KEY

Deprecated.

See Also:

Constant Field Values

ALWAYS_COPY_FILES

public static final String ALWAYS_COPY_FILES

Deprecated.

See Also:

Constant Field Values

FAIL_IF_NEED_SPLIT_HFILE

public static final String FAIL_IF_NEED_SPLIT_HFILE

Deprecated.

See Also:

Constant Field Values

TMP_DIR

static final String TMP_DIR

Deprecated.

See Also:

Constant Field Values

maxFilesPerRegionPerFamily

private int maxFilesPerRegionPerFamily

Deprecated.

assignSeqIds

private boolean assignSeqIds

Deprecated.

bulkLoadByFamily

private boolean bulkLoadByFamily

Deprecated.

fsDelegationToken

private FsDelegationToken fsDelegationToken

Deprecated.

userProvider

private UserProvider userProvider

Deprecated.

nrThreads

private int nrThreads

Deprecated.

numRetries

private AtomicInteger numRetries

Deprecated.

rpcControllerFactory

private RpcControllerFactory rpcControllerFactory

Deprecated.

bulkToken

private String bulkToken

Deprecated.

clusterIds

private List<String> clusterIds

Deprecated.

replicate

private boolean replicate

Deprecated.

failIfNeedSplitHFile

private boolean failIfNeedSplitHFile

Deprecated.

Constructor Detail

LoadIncrementalHFiles

public LoadIncrementalHFiles(org.apache.hadoop.conf.Configuration conf)

Deprecated.

Method Detail

initialize

public void initialize()

Deprecated.

usage

private void usage()

Deprecated.

prepareHFileQueue

public void prepareHFileQueue(org.apache.hadoop.fs.Path hfilesDir,
                              Table table,
                              Deque<LoadIncrementalHFiles.LoadQueueItem> queue,
                              boolean validateHFile)
                       throws IOException

Deprecated.

Prepare a collection of LoadIncrementalHFiles.LoadQueueItem from list of source hfiles contained in the passed directory and validates whether the prepared queue has all the valid table column families in it.

Parameters:

hfilesDir - directory containing list of hfiles to be loaded into the table

table - table to which hfiles should be loaded

queue - queue which needs to be loaded into the table

validateHFile - if true hfiles will be validated for its format

Throws:

IOException - If any I/O or network error occurred

prepareHFileQueue

public void prepareHFileQueue(org.apache.hadoop.fs.Path hfilesDir,
                              Table table,
                              Deque<LoadIncrementalHFiles.LoadQueueItem> queue,
                              boolean validateHFile,
                              boolean silence)
                       throws IOException

Deprecated.

Prepare a collection of LoadIncrementalHFiles.LoadQueueItem from list of source hfiles contained in the passed directory and validates whether the prepared queue has all the valid table column families in it.

Parameters:

hfilesDir - directory containing list of hfiles to be loaded into the table

table - table to which hfiles should be loaded

queue - queue which needs to be loaded into the table

validateHFile - if true hfiles will be validated for its format

silence - true to ignore unmatched column families

Throws:

IOException - If any I/O or network error occurred

prepareHFileQueue

public void prepareHFileQueue(Map<byte[],List<org.apache.hadoop.fs.Path>> map,
                              Table table,
                              Deque<LoadIncrementalHFiles.LoadQueueItem> queue,
                              boolean silence)
                       throws IOException

Deprecated.

Prepare a collection of LoadIncrementalHFiles.LoadQueueItem from list of source hfiles contained in the passed directory and validates whether the prepared queue has all the valid table column families in it.

Parameters:

map - map of family to List of hfiles

table - table to which hfiles should be loaded

queue - queue which needs to be loaded into the table

silence - true to ignore unmatched column families

Throws:

IOException - If any I/O or network error occurred

doBulkLoad

public Map<LoadIncrementalHFiles.LoadQueueItem,ByteBuffer> doBulkLoad(org.apache.hadoop.fs.Path hfofDir,
                                                                      Admin admin,
                                                                      Table table,
                                                                      RegionLocator regionLocator)
                                                               throws TableNotFoundException,
                                                                      IOException

Deprecated.

Perform a bulk load of the given directory into the given pre-existing table. This method is not threadsafe.

Parameters:

hfofDir - the directory that was provided as the output path of a job using HFileOutputFormat

admin - the Admin

table - the table to load into

regionLocator - region locator

Throws:

TableNotFoundException - if table does not yet exist

IOException

doBulkLoad

public Map<LoadIncrementalHFiles.LoadQueueItem,ByteBuffer> doBulkLoad(Map<byte[],List<org.apache.hadoop.fs.Path>> map,
                                                                      Admin admin,
                                                                      Table table,
                                                                      RegionLocator regionLocator,
                                                                      boolean silence,
                                                                      boolean copyFile)
                                                               throws TableNotFoundException,
                                                                      IOException

Deprecated.

Perform a bulk load of the given directory into the given pre-existing table. This method is not threadsafe.

Parameters:

map - map of family to List of hfiles

admin - the Admin

table - the table to load into

regionLocator - region locator

silence - true to ignore unmatched column families

copyFile - always copy hfiles if true

Throws:

TableNotFoundException - if table does not yet exist

IOException

doBulkLoad

public Map<LoadIncrementalHFiles.LoadQueueItem,ByteBuffer> doBulkLoad(org.apache.hadoop.fs.Path hfofDir,
                                                                      Admin admin,
                                                                      Table table,
                                                                      RegionLocator regionLocator,
                                                                      boolean silence,
                                                                      boolean copyFile)
                                                               throws TableNotFoundException,
                                                                      IOException

Deprecated.

Perform a bulk load of the given directory into the given pre-existing table. This method is not threadsafe.

Parameters:

hfofDir - the directory that was provided as the output path of a job using HFileOutputFormat

admin - the Admin

table - the table to load into

regionLocator - region locator

silence - true to ignore unmatched column families

copyFile - always copy hfiles if true

Throws:

TableNotFoundException - if table does not yet exist

IOException

loadHFileQueue

public void loadHFileQueue(Table table,
                           Connection conn,
                           Deque<LoadIncrementalHFiles.LoadQueueItem> queue,
                           Pair<byte[][],byte[][]> startEndKeys)
                    throws IOException

Deprecated.

Used by the replication sink to load the hfiles from the source cluster. It does the following,

1. LoadIncrementalHFiles#groupOrSplitPhase(Table, ExecutorService, Deque, Pair)}
2. LoadIncrementalHFiles#bulkLoadPhase(Table, Connection, ExecutorService, Deque, Multimap)

Parameters:

table - Table to which these hfiles should be loaded to

conn - Connection to use

queue - LoadIncrementalHFiles.LoadQueueItem has hfiles yet to be loaded

startEndKeys - starting and ending row keys of the region

Throws:

IOException

loadHFileQueue

public void loadHFileQueue(Table table,
                           Connection conn,
                           Deque<LoadIncrementalHFiles.LoadQueueItem> queue,
                           Pair<byte[][],byte[][]> startEndKeys,
                           boolean copyFile)
                    throws IOException

Deprecated.

Used by the replication sink to load the hfiles from the source cluster. It does the following,

1. LoadIncrementalHFiles#groupOrSplitPhase(Table, ExecutorService, Deque, Pair)}
2. LoadIncrementalHFiles#bulkLoadPhase(Table, Connection, ExecutorService, Deque, Multimap)

Parameters:

table - Table to which these hfiles should be loaded to

conn - Connection to use

queue - LoadIncrementalHFiles.LoadQueueItem has hfiles yet to be loaded

startEndKeys - starting and ending row keys of the region

Throws:

IOException

performBulkLoad

private Map<LoadIncrementalHFiles.LoadQueueItem,ByteBuffer> performBulkLoad(Admin admin,
                                                                            Table table,
                                                                            RegionLocator regionLocator,
                                                                            Deque<LoadIncrementalHFiles.LoadQueueItem> queue,
                                                                            ExecutorService pool,
                                                                            SecureBulkLoadClient secureClient,
                                                                            boolean copyFile)
                                                                     throws IOException

Deprecated.

Throws:

IOException

groupByFamilies

private Map<byte[],Collection<LoadIncrementalHFiles.LoadQueueItem>> groupByFamilies(Collection<LoadIncrementalHFiles.LoadQueueItem> itemsInRegion)

Deprecated.

bulkLoadPhase

@InterfaceAudience.Private
protected void bulkLoadPhase(Table table,
                                                        Connection conn,
                                                        ExecutorService pool,
                                                        Deque<LoadIncrementalHFiles.LoadQueueItem> queue,
                                                        org.apache.hbase.thirdparty.com.google.common.collect.Multimap<ByteBuffer,LoadIncrementalHFiles.LoadQueueItem> regionGroups,
                                                        boolean copyFile,
                                                        Map<LoadIncrementalHFiles.LoadQueueItem,ByteBuffer> item2RegionMap)
                                                 throws IOException

Deprecated.

This takes the LQI's grouped by likely regions and attempts to bulk load them. Any failures are re-queued for another pass with the groupOrSplitPhase.

protected for testing.

Throws:

IOException

buildClientServiceCallable

@InterfaceAudience.Private
protected ClientServiceCallable<byte[]> buildClientServiceCallable(Connection conn,
                                                                                              TableName tableName,
                                                                                              byte[] first,
                                                                                              Collection<LoadIncrementalHFiles.LoadQueueItem> lqis,
                                                                                              boolean copyFile)

Deprecated.

checkHFilesCountPerRegionPerFamily

private boolean checkHFilesCountPerRegionPerFamily(org.apache.hbase.thirdparty.com.google.common.collect.Multimap<ByteBuffer,LoadIncrementalHFiles.LoadQueueItem> regionGroups)

Deprecated.

groupOrSplitPhase

private Pair<org.apache.hbase.thirdparty.com.google.common.collect.Multimap<ByteBuffer,LoadIncrementalHFiles.LoadQueueItem>,Set<String>> groupOrSplitPhase(Table table,
                                                                                                                                                           ExecutorService pool,
                                                                                                                                                           Deque<LoadIncrementalHFiles.LoadQueueItem> queue,
                                                                                                                                                           Pair<byte[][],byte[][]> startEndKeys)
                                                                                                                                                    throws IOException

Deprecated.

Parameters:

table - the table to load into

pool - the ExecutorService

queue - the queue for LoadQueueItem

startEndKeys - start and end keys

Returns:

A map that groups LQI by likely bulk load region targets and Set of missing hfiles.

Throws:

IOException

splitStoreFile

private List<LoadIncrementalHFiles.LoadQueueItem> splitStoreFile(LoadIncrementalHFiles.LoadQueueItem item,
                                                                 Table table,
                                                                 byte[] startKey,
                                                                 byte[] splitKey)
                                                          throws IOException

Deprecated.

Throws:

IOException

getRegionIndex

private int getRegionIndex(Pair<byte[][],byte[][]> startEndKeys,
                           byte[] key)

Deprecated.

Parameters:

startEndKeys - the start/end keys of regions belong to this table, the list in ascending order by start key

key - the key need to find which region belong to

Returns:

region index

checkRegionIndexValid

private void checkRegionIndexValid(int idx,
                                   Pair<byte[][],byte[][]> startEndKeys,
                                   TableName tableName)
                            throws IOException

Deprecated.

we can consider there is a region hole in following conditions. 1) if idx < 0,then first region info is lost. 2) if the endkey of a region is not equal to the startkey of the next region. 3) if the endkey of the last region is not empty.

Throws:

IOException

groupOrSplit

@InterfaceAudience.Private
protected Pair<List<LoadIncrementalHFiles.LoadQueueItem>,String> groupOrSplit(org.apache.hbase.thirdparty.com.google.common.collect.Multimap<ByteBuffer,LoadIncrementalHFiles.LoadQueueItem> regionGroups,
                                                                                                         LoadIncrementalHFiles.LoadQueueItem item,
                                                                                                         Table table,
                                                                                                         Pair<byte[][],byte[][]> startEndKeys)
                                                                                                  throws IOException

Deprecated.

Attempt to assign the given load queue item into its target region group. If the hfile boundary no longer fits into a region, physically splits the hfile such that the new bottom half will fit and returns the list of LQI's corresponding to the resultant hfiles.

protected for testing

Throws:

IOException - if an IO failure is encountered

tryAtomicRegionLoad

@Deprecated
 @InterfaceAudience.Private
protected List<LoadIncrementalHFiles.LoadQueueItem> tryAtomicRegionLoad(Connection conn,
                                                                                                                TableName tableName,
                                                                                                                byte[] first,
                                                                                                                Collection<LoadIncrementalHFiles.LoadQueueItem> lqis,
                                                                                                                boolean copyFile)
                                                                                                         throws IOException

Deprecated. as of release 2.3.0. Use BulkLoadHFiles instead.

Attempts to do an atomic load of many hfiles into a region. If it fails, it returns a list of hfiles that need to be retried. If it is successful it will return an empty list.

NOTE: To maintain row atomicity guarantees, region server callable should succeed atomically and fails atomically.

Protected for testing.

Returns:

empty list if success, list of items to retry on recoverable failure

Throws:

IOException

tryAtomicRegionLoad

@Deprecated
 @InterfaceAudience.Private
protected List<LoadIncrementalHFiles.LoadQueueItem> tryAtomicRegionLoad(ClientServiceCallable<byte[]> serviceCallable,
                                                                                                                TableName tableName,
                                                                                                                byte[] first,
                                                                                                                Collection<LoadIncrementalHFiles.LoadQueueItem> lqis)
                                                                                                         throws IOException

Deprecated. as of release 2.3.0. Use BulkLoadHFiles instead.

Attempts to do an atomic load of many hfiles into a region. If it fails, it returns a list of hfiles that need to be retried. If it is successful it will return an empty list.

NOTE: To maintain row atomicity guarantees, region server callable should succeed atomically and fails atomically.

Protected for testing.

Returns:

empty list if success, list of items to retry on recoverable failure

Throws:

IOException

createTable

private void createTable(TableName tableName,
                         org.apache.hadoop.fs.Path hfofDir,
                         Admin admin)
                  throws IOException

Deprecated.

If the table is created for the first time, then "completebulkload" reads the files twice. More modifications necessary if we want to avoid doing it.

Throws:

IOException

cleanup

private void cleanup(Admin admin,
                     Deque<LoadIncrementalHFiles.LoadQueueItem> queue,
                     ExecutorService pool,
                     SecureBulkLoadClient secureClient)
              throws IOException

Deprecated.

Throws:

IOException

getUniqueName

private String getUniqueName()

Deprecated.

validateFamiliesInHFiles

private void validateFamiliesInHFiles(Table table,
                                      Deque<LoadIncrementalHFiles.LoadQueueItem> queue,
                                      boolean silence)
                               throws IOException

Deprecated.

Checks whether there is any invalid family name in HFiles to be bulk loaded.

Throws:

IOException

populateLoadQueue

private void populateLoadQueue(Deque<LoadIncrementalHFiles.LoadQueueItem> ret,
                               Map<byte[],List<org.apache.hadoop.fs.Path>> map)

Deprecated.

Populate the Queue with given HFiles

discoverLoadQueue

private void discoverLoadQueue(Deque<LoadIncrementalHFiles.LoadQueueItem> ret,
                               org.apache.hadoop.fs.Path hfofDir,
                               boolean validateHFile)
                        throws IOException

Deprecated.

Walk the given directory for all HFiles, and return a Queue containing all such files.

Throws:

IOException

visitBulkHFiles

private static <TFamily> void visitBulkHFiles(org.apache.hadoop.fs.FileSystem fs,
                                              org.apache.hadoop.fs.Path bulkDir,
                                              LoadIncrementalHFiles.BulkHFileVisitor<TFamily> visitor)
                                       throws IOException

Deprecated.

Iterate over the bulkDir hfiles. Skip reference, HFileLink, files starting with "_" and non-valid hfiles.

Throws:

IOException

visitBulkHFiles

private static <TFamily> void visitBulkHFiles(org.apache.hadoop.fs.FileSystem fs,
                                              org.apache.hadoop.fs.Path bulkDir,
                                              LoadIncrementalHFiles.BulkHFileVisitor<TFamily> visitor,
                                              boolean validateHFile)
                                       throws IOException

Deprecated.

Iterate over the bulkDir hfiles. Skip reference, HFileLink, files starting with "_". Check and skip non-valid hfiles by default, or skip this validation by setting 'hbase.loadincremental.validate.hfile' to false.

Throws:

IOException

createExecutorService

private ExecutorService createExecutorService()

Deprecated.

toString

private final String toString(List<Pair<byte[],String>> list)

Deprecated.

isSecureBulkLoadEndpointAvailable

private boolean isSecureBulkLoadEndpointAvailable()

Deprecated.

splitStoreFile

@InterfaceAudience.Private
static void splitStoreFile(org.apache.hadoop.conf.Configuration conf,
                                                      org.apache.hadoop.fs.Path inFile,
                                                      ColumnFamilyDescriptor familyDesc,
                                                      byte[] splitKey,
                                                      org.apache.hadoop.fs.Path bottomOut,
                                                      org.apache.hadoop.fs.Path topOut)
                                               throws IOException

Deprecated.

Split a storefile into a top and bottom half, maintaining the metadata, recreating bloom filters, etc.

Throws:

IOException

copyHFileHalf

private static void copyHFileHalf(org.apache.hadoop.conf.Configuration conf,
                                  org.apache.hadoop.fs.Path inFile,
                                  org.apache.hadoop.fs.Path outFile,
                                  Reference reference,
                                  ColumnFamilyDescriptor familyDescriptor)
                           throws IOException

Deprecated.

Copy half of an HFile into a new HFile.

Throws:

IOException

shouldCopyHFileMetaKey

private static boolean shouldCopyHFileMetaKey(byte[] key)

Deprecated.

isCreateTable

private boolean isCreateTable()

Deprecated.

isSilence

private boolean isSilence()

Deprecated.

isAlwaysCopyFiles

private boolean isAlwaysCopyFiles()

Deprecated.

run

protected final Map<LoadIncrementalHFiles.LoadQueueItem,ByteBuffer> run(org.apache.hadoop.fs.Path hfofDir,
                                                                        TableName tableName)
                                                                 throws IOException

Deprecated.

Throws:

IOException

run

public Map<LoadIncrementalHFiles.LoadQueueItem,ByteBuffer> run(String hfofDir,
                                                               TableName tableName)
                                                        throws IOException

Deprecated.

Perform bulk load on the given table.

Parameters:

hfofDir - the directory that was provided as the output path of a job using HFileOutputFormat

tableName - the table to load into

Throws:

IOException

run

public Map<LoadIncrementalHFiles.LoadQueueItem,ByteBuffer> run(Map<byte[],List<org.apache.hadoop.fs.Path>> family2Files,
                                                               TableName tableName)
                                                        throws IOException

Deprecated.

Perform bulk load on the given table.

Parameters:

family2Files - map of family to List of hfiles

tableName - the table to load into

Throws:

IOException

run

public int run(String[] args)
        throws Exception

Deprecated.

Specified by:

run in interface org.apache.hadoop.util.Tool

Throws:

Exception

main

public static void main(String[] args)
                 throws Exception

Deprecated.

Throws:

Exception

setBulkToken

public void setBulkToken(String stagingDir)

Deprecated.

Called from replication sink, where it manages bulkToken(staging directory) by itself. This is used only when SecureBulkLoadEndpoint is configured in hbase.coprocessor.region.classes property. This directory is used as a temporary directory where all files are initially copied/moved from user given directory, set all the required file permissions and then from their it is finally loaded into a table. This should be set only when, one would like to manage the staging directory by itself. Otherwise this tool will handle this by itself.

Parameters:

stagingDir - staging directory path

setClusterIds

public void setClusterIds(List<String> clusterIds)

Deprecated.

disableReplication

public void disableReplication()

Deprecated.

Disables replication for these bulkloaded files.

inferBoundaries

public static byte[][] inferBoundaries(SortedMap<byte[],Integer> bdryMap)

Deprecated.

Infers region boundaries for a new table.

Parameter:
bdryMap is a map between keys to an integer belonging to {+1, -1}

• If a key is a start key of a file, then it maps to +1
• If a key is an end key of a file, then it maps to -1

Algo:

1. Poll on the keys in order:
   1. Keep adding the mapped values to these keys (runningSum)
   2. Each time runningSum reaches 0, add the start Key from when the runningSum had started to a boundary list.
2. Return the boundary list.