org.apache.hadoop.hbase.tool

Class TestLoadIncrementalHFilesSplitRecovery

java.lang.Object

org.apache.hadoop.hbase.tool.TestLoadIncrementalHFilesSplitRecovery

Direct Known Subclasses:

TestSecureLoadIncrementalHFilesSplitRecovery

public class TestLoadIncrementalHFilesSplitRecovery
extends Object

Test cases for the atomic load error handling of the bulk load functionality.

Field Summary

Fields

Modifier and Type	Field and Description
static HBaseClassTestRule	CLASS_RULE
private static byte[][]	families
private static org.slf4j.Logger	LOG
org.junit.rules.TestName	name
(package private) static int	NUM_CFS
(package private) static byte[]	QUAL
(package private) static int	ROWCOUNT
(package private) static boolean	useSecure
(package private) static HBaseTestingUtility	util

Constructor Summary

Constructors

Constructor and Description
TestLoadIncrementalHFilesSplitRecovery()

Method Summary

Modifier and Type	Method and Description
(package private) void	assertExpectedTable(org.apache.hadoop.hbase.client.Connection connection, org.apache.hadoop.hbase.TableName table, int count, int value) Checks that all columns have the expected value and that there is the expected number of rows.
(package private) void	assertExpectedTable(org.apache.hadoop.hbase.TableName table, int count, int value) Checks that all columns have the expected value and that there is the expected number of rows.
private org.apache.hadoop.fs.Path	buildBulkFiles(org.apache.hadoop.hbase.TableName table, int value)
static void	buildHFiles(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path dir, int value)
private org.apache.hadoop.hbase.client.TableDescriptor	createTableDesc(org.apache.hadoop.hbase.TableName name, int cfs)
(package private) static String	family(int i)
private void	forceSplit(org.apache.hadoop.hbase.TableName table) Split the known table in half.
private org.apache.hadoop.hbase.client.ClusterConnection	getMockedConnection(org.apache.hadoop.conf.Configuration conf)
private void	populateTable(org.apache.hadoop.hbase.client.Connection connection, org.apache.hadoop.hbase.TableName table, int value) Populate table with known values.
(package private) static byte[]	rowkey(int i)
static void	setupCluster()
private void	setupTable(org.apache.hadoop.hbase.client.Connection connection, org.apache.hadoop.hbase.TableName table, int cfs) Creates a table with given table name and specified number of column families if the table does not already exist.
private void	setupTableWithSplitkeys(org.apache.hadoop.hbase.TableName table, int cfs, byte[][] SPLIT_KEYS) Creates a table with given table name,specified number of column families and splitkeys if the table does not already exist.
static void	teardownCluster()
void	testBulkLoadPhaseFailure() Test that shows that exception thrown from the RS side will result in an exception on the LIHFile client.
void	testGroupOrSplitFailure() This simulates an remote exception which should cause LIHF to exit with an exception.
void	testGroupOrSplitPresplit() This test splits a table and attempts to bulk load.
void	testGroupOrSplitWhenRegionHoleExistsInMeta()
void	testRetryOnIOException() Test that shows that exception thrown from the RS side will result in the expected number of retries set by $HConstants.HBASE_CLIENT_RETRIES_NUMBER when $LoadIncrementalHFiles.RETRY_ON_IO_EXCEPTION is set
void	testSplitTmpFileCleanUp() This test creates a table with many small regions.
void	testSplitWhileBulkLoadPhase() This test exercises the path where there is a split after initial validation but before the atomic bulk load call.
(package private) static byte[]	value(int i)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

CLASS_RULE

public static final HBaseClassTestRule CLASS_RULE

LOG

private static final org.slf4j.Logger LOG

util

static HBaseTestingUtility util

useSecure

static boolean useSecure

NUM_CFS

static final int NUM_CFS

See Also:

Constant Field Values

QUAL

static final byte[] QUAL

ROWCOUNT

static final int ROWCOUNT

See Also:

Constant Field Values

families

private static final byte[][] families

name

public org.junit.rules.TestName name

Constructor Detail

TestLoadIncrementalHFilesSplitRecovery

public TestLoadIncrementalHFilesSplitRecovery()

Method Detail

rowkey

static byte[] rowkey(int i)

family

static String family(int i)

value

static byte[] value(int i)

buildHFiles

public static void buildHFiles(org.apache.hadoop.fs.FileSystem fs,
                               org.apache.hadoop.fs.Path dir,
                               int value)
                        throws IOException

Throws:

IOException

createTableDesc

private org.apache.hadoop.hbase.client.TableDescriptor createTableDesc(org.apache.hadoop.hbase.TableName name,
                                                                       int cfs)

setupTable

private void setupTable(org.apache.hadoop.hbase.client.Connection connection,
                        org.apache.hadoop.hbase.TableName table,
                        int cfs)
                 throws IOException

Creates a table with given table name and specified number of column families if the table does not already exist.

Throws:

IOException

setupTableWithSplitkeys

private void setupTableWithSplitkeys(org.apache.hadoop.hbase.TableName table,
                                     int cfs,
                                     byte[][] SPLIT_KEYS)
                              throws IOException

Creates a table with given table name,specified number of column families
and splitkeys if the table does not already exist.

Parameters:

table -

cfs -

SPLIT_KEYS -

Throws:

IOException

buildBulkFiles

private org.apache.hadoop.fs.Path buildBulkFiles(org.apache.hadoop.hbase.TableName table,
                                                 int value)
                                          throws Exception

Throws:

Exception

populateTable

private void populateTable(org.apache.hadoop.hbase.client.Connection connection,
                           org.apache.hadoop.hbase.TableName table,
                           int value)
                    throws Exception

Populate table with known values.

Throws:

Exception

forceSplit

private void forceSplit(org.apache.hadoop.hbase.TableName table)

Split the known table in half. (this is hard coded for this test suite)

setupCluster

public static void setupCluster()
                         throws Exception

Throws:

Exception

teardownCluster

public static void teardownCluster()
                            throws Exception

Throws:

Exception

assertExpectedTable

void assertExpectedTable(org.apache.hadoop.hbase.TableName table,
                         int count,
                         int value)
                  throws IOException

Checks that all columns have the expected value and that there is the expected number of rows.

Throws:

IOException

testBulkLoadPhaseFailure

public void testBulkLoadPhaseFailure()
                              throws Exception

Test that shows that exception thrown from the RS side will result in an exception on the LIHFile client.

Throws:

Exception

testRetryOnIOException

public void testRetryOnIOException()
                            throws Exception

Test that shows that exception thrown from the RS side will result in the expected number of retries set by $HConstants.HBASE_CLIENT_RETRIES_NUMBER when $LoadIncrementalHFiles.RETRY_ON_IO_EXCEPTION is set

Throws:

Exception

getMockedConnection

private org.apache.hadoop.hbase.client.ClusterConnection getMockedConnection(org.apache.hadoop.conf.Configuration conf)
                                                                      throws IOException,
                                                                             org.apache.hbase.thirdparty.com.google.protobuf.ServiceException

Throws:

IOException

org.apache.hbase.thirdparty.com.google.protobuf.ServiceException

testSplitWhileBulkLoadPhase

public void testSplitWhileBulkLoadPhase()
                                 throws Exception

This test exercises the path where there is a split after initial validation but before the atomic bulk load call. We cannot use presplitting to test this path, so we actually inject a split just before the atomic region load.

Throws:

Exception

testGroupOrSplitPresplit

public void testGroupOrSplitPresplit()
                              throws Exception

This test splits a table and attempts to bulk load. The bulk import files should be split before atomically importing.

Throws:

Exception

testSplitTmpFileCleanUp

public void testSplitTmpFileCleanUp()
                             throws Exception

This test creates a table with many small regions. The bulk load files would be splitted multiple times before all of them can be loaded successfully.

Throws:

Exception

testGroupOrSplitFailure

public void testGroupOrSplitFailure()
                             throws Exception

This simulates an remote exception which should cause LIHF to exit with an exception.

Throws:

Exception

testGroupOrSplitWhenRegionHoleExistsInMeta

public void testGroupOrSplitWhenRegionHoleExistsInMeta()
                                                throws Exception

Throws:

Exception

assertExpectedTable

void assertExpectedTable(org.apache.hadoop.hbase.client.Connection connection,
                         org.apache.hadoop.hbase.TableName table,
                         int count,
                         int value)
                  throws IOException

Checks that all columns have the expected value and that there is the expected number of rows.

Throws:

IOException