LoadIncrementalHFiles (Apache HBase 1.2.12 API)

java.lang.Object
- org.apache.hadoop.conf.Configured
- - org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, org.apache.hadoop.util.Tool
```
@InterfaceAudience.Public
@InterfaceStability.Stable
public class LoadIncrementalHFiles
extends org.apache.hadoop.conf.Configured
implements org.apache.hadoop.util.Tool
```
Tool to load the output of HFileOutputFormat into an existing table.

See Also:
usage()

Nested Class Summary

Nested Classes
Modifier and Type	Class and Description
`private static interface`	`LoadIncrementalHFiles.BulkHFileVisitor<TFamily>`
`static class`	`LoadIncrementalHFiles.LoadQueueItem` Represents an HFile waiting to be loaded.

Field Summary

Fields
Modifier and Type	Field and Description
`private static String`	`ASSIGN_SEQ_IDS`
`private boolean`	`assignSeqIds`
`private String`	`bulkToken`
`static String`	`CREATE_TABLE_CONF_KEY`
`private org.apache.hadoop.fs.FileSystem`	`fs`
`private FsDelegationToken`	`fsDelegationToken`
`private Admin`	`hbAdmin`
`private static org.apache.commons.logging.Log`	`LOG`
`static String`	`MAX_FILES_PER_REGION_PER_FAMILY`
`private int`	`maxFilesPerRegionPerFamily`
`static String`	`NAME`
`(package private) static String`	`TMP_DIR`
`private UserProvider`	`userProvider`

Constructor Summary

Constructors
Modifier Constructor and Description

private LoadIncrementalHFiles()

LoadIncrementalHFiles(org.apache.hadoop.conf.Configuration conf)

Constructors
Modifier	Constructor and Description
`private`	`LoadIncrementalHFiles()`
	`LoadIncrementalHFiles(org.apache.hadoop.conf.Configuration conf)`

Method Summary

Methods
Modifier and Type	Method and Description
`protected void`	`bulkLoadPhase(Table table, Connection conn, ExecutorService pool, Deque<LoadIncrementalHFiles.LoadQueueItem> queue, com.google.common.collect.Multimap<ByteBuffer,LoadIncrementalHFiles.LoadQueueItem> regionGroups)` This takes the LQI's grouped by likely regions and attempts to bulk load them.
`private boolean`	`checkHFilesCountPerRegionPerFamily(com.google.common.collect.Multimap<ByteBuffer,LoadIncrementalHFiles.LoadQueueItem> regionGroups)`
`private static void`	`copyHFileHalf(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path inFile, org.apache.hadoop.fs.Path outFile, Reference reference, HColumnDescriptor familyDescriptor)` Copy half of an HFile into a new HFile.
`private void`	`createTable(TableName tableName, String dirPath)`
`private void`	`discoverLoadQueue(Deque<LoadIncrementalHFiles.LoadQueueItem> ret, org.apache.hadoop.fs.Path hfofDir, boolean validateHFile)` Walk the given directory for all HFiles, and return a Queue containing all such files.
`void`	`doBulkLoad(org.apache.hadoop.fs.Path hfofDir, Admin admin, Table table, RegionLocator regionLocator)` Perform a bulk load of the given directory into the given pre-existing table.
`void`	`doBulkLoad(org.apache.hadoop.fs.Path hfofDir, HTable table)` Deprecated. since hbase-1.3.0, hbase-1.2.2, and hbase-1.1.6; replaced by `doBulkLoad(Path, Admin, Table, RegionLocator)`.
`private boolean`	`doesTableExist(TableName tableName)`
`private String`	`getUniqueName()`
`protected List<LoadIncrementalHFiles.LoadQueueItem>`	`groupOrSplit(com.google.common.collect.Multimap<ByteBuffer,LoadIncrementalHFiles.LoadQueueItem> regionGroups, LoadIncrementalHFiles.LoadQueueItem item, Table table, Pair<byte[][],byte[][]> startEndKeys)` Attempt to assign the given load queue item into its target region group.
`private com.google.common.collect.Multimap<ByteBuffer,LoadIncrementalHFiles.LoadQueueItem>`	`groupOrSplitPhase(Table table, ExecutorService pool, Deque<LoadIncrementalHFiles.LoadQueueItem> queue, Pair<byte[][],byte[][]> startEndKeys)`
`static byte[][]`	`inferBoundaries(TreeMap<byte[],Integer> bdryMap)`
`private void`	`initialize()`
`private boolean`	`isSecureBulkLoadEndpointAvailable()`
`static void`	`main(String[] args)`
`int`	`run(String[] args)`
`private static boolean`	`shouldCopyHFileMetaKey(byte[] key)`
`(package private) static void`	`splitStoreFile(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path inFile, HColumnDescriptor familyDesc, byte[] splitKey, org.apache.hadoop.fs.Path bottomOut, org.apache.hadoop.fs.Path topOut)` Split a storefile into a top and bottom half, maintaining the metadata, recreating bloom filters, etc.
`protected List<LoadIncrementalHFiles.LoadQueueItem>`	`splitStoreFile(LoadIncrementalHFiles.LoadQueueItem item, Table table, byte[] startKey, byte[] splitKey)`
`protected List<LoadIncrementalHFiles.LoadQueueItem>`	`tryAtomicRegionLoad(Connection conn, TableName tableName, byte[] first, Collection<LoadIncrementalHFiles.LoadQueueItem> lqis)` Attempts to do an atomic load of many hfiles into a region.
`protected List<LoadIncrementalHFiles.LoadQueueItem>`	`tryAtomicRegionLoad(HConnection conn, byte[] tableName, byte[] first, Collection<LoadIncrementalHFiles.LoadQueueItem> lqis)` Deprecated. As of release 0.96 (HBASE-9508). This will be removed in HBase 2.0.0. Use `tryAtomicRegionLoad(Connection, TableName, byte[], Collection)`.
`private void`	`usage()`
`private static <TFamily> void`	`visitBulkHFiles(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path bulkDir, LoadIncrementalHFiles.BulkHFileVisitor<TFamily> visitor)` Iterate over the bulkDir hfiles.
`private static <TFamily> void`	`visitBulkHFiles(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path bulkDir, LoadIncrementalHFiles.BulkHFileVisitor<TFamily> visitor, boolean validateHFile)` Iterate over the bulkDir hfiles.

Methods inherited from class org.apache.hadoop.conf.Configured
getConf, setConf

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.conf.Configurable
getConf, setConf

Field Detail

LOG

private static final org.apache.commons.logging.Log LOG

hbAdmin
```
private Admin hbAdmin
```

NAME
```
public static final String NAME
```
See Also:
Constant Field Values

MAX_FILES_PER_REGION_PER_FAMILY

public static final String MAX_FILES_PER_REGION_PER_FAMILY

See Also:: Constant Field Values

ASSIGN_SEQ_IDS

private static final String ASSIGN_SEQ_IDS

See Also:: Constant Field Values

CREATE_TABLE_CONF_KEY

public static final String CREATE_TABLE_CONF_KEY

See Also:: Constant Field Values

TMP_DIR
```
static final String TMP_DIR
```
See Also:
Constant Field Values

maxFilesPerRegionPerFamily
```
private int maxFilesPerRegionPerFamily
```

assignSeqIds
```
private boolean assignSeqIds
```

fs

private org.apache.hadoop.fs.FileSystem fs

fsDelegationToken

private FsDelegationToken fsDelegationToken

bulkToken
```
private String bulkToken
```

userProvider
```
private UserProvider userProvider
```

Constructor Detail

LoadIncrementalHFiles
```
private LoadIncrementalHFiles()
```

LoadIncrementalHFiles

public LoadIncrementalHFiles(org.apache.hadoop.conf.Configuration conf)
                      throws Exception

Throws:: Exception

Method Detail

initialize

private void initialize()
                 throws Exception

Throws:: Exception

usage
```
private void usage()
```

visitBulkHFiles

private static <TFamily> void visitBulkHFiles(org.apache.hadoop.fs.FileSystem fs,
                             org.apache.hadoop.fs.Path bulkDir,
                             LoadIncrementalHFiles.BulkHFileVisitor<TFamily> visitor)
                             throws IOException

Iterate over the bulkDir hfiles. Skip reference, HFileLink, files starting with "_" and non-valid hfiles.

Throws:: IOException

visitBulkHFiles

private static <TFamily> void visitBulkHFiles(org.apache.hadoop.fs.FileSystem fs,
                             org.apache.hadoop.fs.Path bulkDir,
                             LoadIncrementalHFiles.BulkHFileVisitor<TFamily> visitor,
                             boolean validateHFile)
                             throws IOException

Iterate over the bulkDir hfiles. Skip reference, HFileLink, files starting with "_". Check and skip non-valid hfiles by default, or skip this validation by setting 'hbase.loadincremental.validate.hfile' to false.

Throws:: IOException

discoverLoadQueue

private void discoverLoadQueue(Deque<LoadIncrementalHFiles.LoadQueueItem> ret,
                     org.apache.hadoop.fs.Path hfofDir,
                     boolean validateHFile)
                        throws IOException

Walk the given directory for all HFiles, and return a Queue containing all such files.

Throws:: IOException

doBulkLoad
```
@Deprecated
public void doBulkLoad(org.apache.hadoop.fs.Path hfofDir,
                         HTable table)
                throws TableNotFoundException,
                       IOException
```
Deprecated. since hbase-1.3.0, hbase-1.2.2, and hbase-1.1.6; replaced by doBulkLoad(Path, Admin, Table, RegionLocator).

Perform a bulk load of the given directory into the given pre-existing table. This method is not threadsafe.

Parameters:
hfofDir - the directory that was provided as the output path of a job using HFileOutputFormat
table - the table to load into

Throws:

TableNotFoundException - if table does not yet exist

IOException
See Also:
HBASE-15876

doBulkLoad
```
public void doBulkLoad(org.apache.hadoop.fs.Path hfofDir,
              Admin admin,
              Table table,
              RegionLocator regionLocator)
                throws TableNotFoundException,
                       IOException
```
Perform a bulk load of the given directory into the given pre-existing table. This method is not threadsafe.

Parameters:
hfofDir - the directory that was provided as the output path of a job using HFileOutputFormat
table - the table to load into

Throws:

TableNotFoundException - if table does not yet exist

IOException

bulkLoadPhase

protected void bulkLoadPhase(Table table,
                 Connection conn,
                 ExecutorService pool,
                 Deque<LoadIncrementalHFiles.LoadQueueItem> queue,
                 com.google.common.collect.Multimap<ByteBuffer,LoadIncrementalHFiles.LoadQueueItem> regionGroups)
                      throws IOException

This takes the LQI's grouped by likely regions and attempts to bulk load them. Any failures are re-queued for another pass with the groupOrSplitPhase.

Throws:: IOException

checkHFilesCountPerRegionPerFamily

private boolean checkHFilesCountPerRegionPerFamily(com.google.common.collect.Multimap<ByteBuffer,LoadIncrementalHFiles.LoadQueueItem> regionGroups)

groupOrSplitPhase

private com.google.common.collect.Multimap<ByteBuffer,LoadIncrementalHFiles.LoadQueueItem> groupOrSplitPhase(Table table,
                                                                                                   ExecutorService pool,
                                                                                                   Deque<LoadIncrementalHFiles.LoadQueueItem> queue,
                                                                                                   Pair<byte[][],byte[][]> startEndKeys)
                                                                                                      throws IOException

Returns:: A Multimap that groups LQI by likely bulk load region targets.
Throws:: IOException

getUniqueName
```
private String getUniqueName()
```

splitStoreFile

protected List<LoadIncrementalHFiles.LoadQueueItem> splitStoreFile(LoadIncrementalHFiles.LoadQueueItem item,
                                                       Table table,
                                                       byte[] startKey,
                                                       byte[] splitKey)
                                                            throws IOException

Throws:: IOException

groupOrSplit

protected List<LoadIncrementalHFiles.LoadQueueItem> groupOrSplit(com.google.common.collect.Multimap<ByteBuffer,LoadIncrementalHFiles.LoadQueueItem> regionGroups,
                                                     LoadIncrementalHFiles.LoadQueueItem item,
                                                     Table table,
                                                     Pair<byte[][],byte[][]> startEndKeys)
                                                          throws IOException

Attempt to assign the given load queue item into its target region group. If the hfile boundary no longer fits into a region, physically splits the hfile such that the new bottom half will fit and returns the list of LQI's corresponding to the resultant hfiles. protected for testing

Throws:: IOException

tryAtomicRegionLoad

@Deprecated
protected List<LoadIncrementalHFiles.LoadQueueItem> tryAtomicRegionLoad(HConnection conn,
                                                                       byte[] tableName,
                                                                       byte[] first,
                                                                       Collection<LoadIncrementalHFiles.LoadQueueItem> lqis)
                                                                 throws IOException

Deprecated. As of release 0.96 (HBASE-9508). This will be removed in HBase 2.0.0. Use tryAtomicRegionLoad(Connection, TableName, byte[], Collection).

Throws:: IOException

tryAtomicRegionLoad

protected List<LoadIncrementalHFiles.LoadQueueItem> tryAtomicRegionLoad(Connection conn,
                                                            TableName tableName,
                                                            byte[] first,
                                                            Collection<LoadIncrementalHFiles.LoadQueueItem> lqis)
                                                                 throws IOException

Attempts to do an atomic load of many hfiles into a region. If it fails, it returns a list of hfiles that need to be retried. If it is successful it will return an empty list. NOTE: To maintain row atomicity guarantees, region server callable should succeed atomically and fails atomically. Protected for testing.

Returns:: empty list if success, list of items to retry on recoverable failure
Throws:: IOException

isSecureBulkLoadEndpointAvailable

private boolean isSecureBulkLoadEndpointAvailable()

splitStoreFile

static void splitStoreFile(org.apache.hadoop.conf.Configuration conf,
                  org.apache.hadoop.fs.Path inFile,
                  HColumnDescriptor familyDesc,
                  byte[] splitKey,
                  org.apache.hadoop.fs.Path bottomOut,
                  org.apache.hadoop.fs.Path topOut)
                    throws IOException

Split a storefile into a top and bottom half, maintaining the metadata, recreating bloom filters, etc.

Throws:: IOException

copyHFileHalf

private static void copyHFileHalf(org.apache.hadoop.conf.Configuration conf,
                 org.apache.hadoop.fs.Path inFile,
                 org.apache.hadoop.fs.Path outFile,
                 Reference reference,
                 HColumnDescriptor familyDescriptor)
                           throws IOException

Copy half of an HFile into a new HFile.

Throws:: IOException

shouldCopyHFileMetaKey

private static boolean shouldCopyHFileMetaKey(byte[] key)

doesTableExist

private boolean doesTableExist(TableName tableName)
                        throws Exception

Throws:: Exception

inferBoundaries

public static byte[][] inferBoundaries(TreeMap<byte[],Integer> bdryMap)

createTable

private void createTable(TableName tableName,
               String dirPath)
                  throws Exception

Throws:: Exception

run
```
public int run(String[] args)
        throws Exception
```
Specified by:

run in interface org.apache.hadoop.util.Tool

Throws:

Exception

main

public static void main(String[] args)
                 throws Exception

Throws:: Exception

Class LoadIncrementalHFiles

Nested Class Summary

Field Summary

Constructor Summary

Method Summary

Methods inherited from class org.apache.hadoop.conf.Configured

Methods inherited from class java.lang.Object

Methods inherited from interface org.apache.hadoop.conf.Configurable

Field Detail

LOG

hbAdmin

NAME

MAX_FILES_PER_REGION_PER_FAMILY

ASSIGN_SEQ_IDS

CREATE_TABLE_CONF_KEY

TMP_DIR

maxFilesPerRegionPerFamily

assignSeqIds

fs

fsDelegationToken

bulkToken

userProvider

Constructor Detail

LoadIncrementalHFiles

LoadIncrementalHFiles

Method Detail

initialize

usage

visitBulkHFiles

visitBulkHFiles

discoverLoadQueue

doBulkLoad

doBulkLoad

bulkLoadPhase

checkHFilesCountPerRegionPerFamily

groupOrSplitPhase

getUniqueName

splitStoreFile

groupOrSplit

tryAtomicRegionLoad

tryAtomicRegionLoad

isSecureBulkLoadEndpointAvailable

splitStoreFile

copyHFileHalf

shouldCopyHFileMetaKey

doesTableExist

inferBoundaries

createTable

run

main