Package org.apache.hadoop.hbase.tool

Class BulkLoadHFilesSplitRecoveryTestBase

java.lang.Object

org.apache.hadoop.hbase.tool.BulkLoadHFilesSplitRecoveryTestBase

Direct Known Subclasses:

SecureBulkLoadHFilesSplitRecoveryTest, TestBulkLoadHFilesSplitRecovery

public class BulkLoadHFilesSplitRecoveryTestBase
extends Object

Test cases for the atomic load error handling of the bulk load functionality.

Field Summary

Fields

Modifier and Type	Field	Description
private static final byte[][]	families
private static final org.slf4j.Logger	LOG
(package private) static final int	NUM_CFS
(package private) static final byte[]	QUAL
(package private) static final int	ROWCOUNT
(package private) static boolean	useSecure
(package private) static HBaseTestingUtil	util

Constructor Summary

Constructors

Constructor	Description
BulkLoadHFilesSplitRecoveryTestBase()

Method Summary

Modifier and Type	Method	Description
(package private) void	assertExpectedTable(org.apache.hadoop.hbase.client.Connection connection, org.apache.hadoop.hbase.TableName table, int count, int value)	Checks that all columns have the expected value and that there is the expected number of rows.
(package private) void	assertExpectedTable(org.apache.hadoop.hbase.TableName table, int count, int value)	Checks that all columns have the expected value and that there is the expected number of rows.
private org.apache.hadoop.fs.Path	buildBulkFiles(org.apache.hadoop.hbase.TableName table, int value)
static void	buildHFiles(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path dir, int value)
private org.apache.hadoop.hbase.client.TableDescriptor	createTableDesc(org.apache.hadoop.hbase.TableName name, int cfs)
private static <T> CompletableFuture<T>	failedFuture(Throwable error)
(package private) static String	family(int i)
private void	forceSplit(org.apache.hadoop.hbase.TableName table)	Split the known table in half.
private static org.apache.hadoop.hbase.client.AsyncClusterConnection	mockAndInjectError(org.apache.hadoop.hbase.client.AsyncClusterConnection conn)
private void	populateTable(org.apache.hadoop.hbase.client.Connection connection, org.apache.hadoop.hbase.TableName table, int value)	Populate table with known values.
(package private) static byte[]	rowkey(int i)
private void	setupTable(org.apache.hadoop.hbase.client.Connection connection, org.apache.hadoop.hbase.TableName table, int cfs)	Creates a table with given table name and specified number of column families if the table does not already exist.
private void	setupTableWithSplitkeys(org.apache.hadoop.hbase.TableName table, int cfs, byte[][] SPLIT_KEYS)	Creates a table with given table name,specified number of column families and splitkeys if the table does not already exist.
static void	teardownCluster()
void	testBulkLoadPhaseFailure(org.junit.jupiter.api.TestInfo testInfo)	Test that shows that exception thrown from the RS side will result in an exception on the LIHFile client.
void	testCorrectSplitPoint(org.junit.jupiter.api.TestInfo testInfo)
void	testGroupOrSplitFailure(org.junit.jupiter.api.TestInfo testInfo)	This simulates an remote exception which should cause LIHF to exit with an exception.
void	testGroupOrSplitPresplit(org.junit.jupiter.api.TestInfo testInfo)	This test splits a table and attempts to bulk load.
void	testRetryOnIOException(org.junit.jupiter.api.TestInfo testInfo)	Test that shows that exception thrown from the RS side will result in the expected number of retries set by $HConstants.HBASE_CLIENT_RETRIES_NUMBER when $BulkLoadHFiles.RETRY_ON_IO_EXCEPTION is set
void	testSplitTmpFileCleanUp(org.junit.jupiter.api.TestInfo testInfo)	This test creates a table with many small regions.
void	testSplitWhileBulkLoadPhase(org.junit.jupiter.api.TestInfo testInfo)	This test exercises the path where there is a split after initial validation but before the atomic bulk load call.
void	testSplitWhileBulkLoadPhaseWithoutItemMap(org.junit.jupiter.api.TestInfo testInfo)	We are testing a split after initial validation but before the atomic bulk load call.
(package private) static byte[]	value(int i)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

LOG

private static final org.slf4j.Logger LOG

util

static HBaseTestingUtil util

useSecure

static boolean useSecure

NUM_CFS

static final int NUM_CFS

See Also:

• Constant Field Values

QUAL

static final byte[] QUAL

ROWCOUNT

static final int ROWCOUNT

See Also:

• Constant Field Values

families

private static final byte[][] families

Constructor Details

BulkLoadHFilesSplitRecoveryTestBase

public BulkLoadHFilesSplitRecoveryTestBase()

Method Details

rowkey

static byte[] rowkey(int i)

family

static String family(int i)

value

static byte[] value(int i)

buildHFiles

public static void buildHFiles(org.apache.hadoop.fs.FileSystem fs,
 org.apache.hadoop.fs.Path dir,
 int value)
                        throws IOException

Throws:

IOException

createTableDesc

private org.apache.hadoop.hbase.client.TableDescriptor createTableDesc(org.apache.hadoop.hbase.TableName name,
 int cfs)

setupTable

private void setupTable(org.apache.hadoop.hbase.client.Connection connection,
 org.apache.hadoop.hbase.TableName table,
 int cfs)
                 throws IOException

Creates a table with given table name and specified number of column families if the table does not already exist.

Throws:

IOException

setupTableWithSplitkeys

private void setupTableWithSplitkeys(org.apache.hadoop.hbase.TableName table,
 int cfs,
 byte[][] SPLIT_KEYS)
                              throws IOException

Creates a table with given table name,specified number of column families
and splitkeys if the table does not already exist.

Throws:

IOException

buildBulkFiles

private org.apache.hadoop.fs.Path buildBulkFiles(org.apache.hadoop.hbase.TableName table,
 int value)
                                          throws Exception

Throws:

Exception

populateTable

private void populateTable(org.apache.hadoop.hbase.client.Connection connection,
 org.apache.hadoop.hbase.TableName table,
 int value)
                    throws Exception

Populate table with known values.

Throws:

Exception

forceSplit

private void forceSplit(org.apache.hadoop.hbase.TableName table)

Split the known table in half. (this is hard coded for this test suite)

teardownCluster

@AfterAll
public static void teardownCluster()
                            throws Exception

Throws:

Exception

assertExpectedTable

void assertExpectedTable(org.apache.hadoop.hbase.TableName table,
 int count,
 int value)
                  throws IOException

Checks that all columns have the expected value and that there is the expected number of rows.

Throws:

IOException

failedFuture

private static <T> CompletableFuture<T> failedFuture(Throwable error)

mockAndInjectError

private static org.apache.hadoop.hbase.client.AsyncClusterConnection mockAndInjectError(org.apache.hadoop.hbase.client.AsyncClusterConnection conn)

testBulkLoadPhaseFailure

@Test
public void testBulkLoadPhaseFailure(org.junit.jupiter.api.TestInfo testInfo)
                              throws Exception

Test that shows that exception thrown from the RS side will result in an exception on the LIHFile client.

Throws:

Exception

testRetryOnIOException

@Test
public void testRetryOnIOException(org.junit.jupiter.api.TestInfo testInfo)
                            throws Exception

Test that shows that exception thrown from the RS side will result in the expected number of retries set by $HConstants.HBASE_CLIENT_RETRIES_NUMBER when $BulkLoadHFiles.RETRY_ON_IO_EXCEPTION is set

Throws:

Exception

testSplitWhileBulkLoadPhase

@Test
public void testSplitWhileBulkLoadPhase(org.junit.jupiter.api.TestInfo testInfo)
                                 throws Exception

This test exercises the path where there is a split after initial validation but before the atomic bulk load call. We cannot use presplitting to test this path, so we actually inject a split just before the atomic region load.

Throws:

Exception

testGroupOrSplitPresplit

@Test
public void testGroupOrSplitPresplit(org.junit.jupiter.api.TestInfo testInfo)
                              throws Exception

This test splits a table and attempts to bulk load. The bulk import files should be split before atomically importing.

Throws:

Exception

testCorrectSplitPoint

@Test
public void testCorrectSplitPoint(org.junit.jupiter.api.TestInfo testInfo)
                           throws Exception

Throws:

Exception

testSplitTmpFileCleanUp

@Test
public void testSplitTmpFileCleanUp(org.junit.jupiter.api.TestInfo testInfo)
                             throws Exception

This test creates a table with many small regions. The bulk load files would be splitted multiple times before all of them can be loaded successfully.

Throws:

Exception

testGroupOrSplitFailure

@Test
public void testGroupOrSplitFailure(org.junit.jupiter.api.TestInfo testInfo)
                             throws Exception

This simulates an remote exception which should cause LIHF to exit with an exception.

Throws:

Exception

testSplitWhileBulkLoadPhaseWithoutItemMap

@Test
public void testSplitWhileBulkLoadPhaseWithoutItemMap(org.junit.jupiter.api.TestInfo testInfo)
                                               throws Exception

We are testing a split after initial validation but before the atomic bulk load call. We cannot use presplitting to test this path, so we actually inject a split just before the atomic region load. However, we will pass null item2RegionMap and that should not affect the bulk load behavior.

Throws:

Exception

assertExpectedTable

void assertExpectedTable(org.apache.hadoop.hbase.client.Connection connection,
 org.apache.hadoop.hbase.TableName table,
 int count,
 int value)
                  throws IOException

Checks that all columns have the expected value and that there is the expected number of rows.

Throws:

IOException