Package org.apache.hadoop.hbase.mapreduce

Class TestImportTsv

java.lang.Object

org.apache.hadoop.hbase.mapreduce.TestImportTsv

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable

@Tag("org.apache.hadoop.hbase.testclassification.VerySlowMapReduceTests") @Tag("org.apache.hadoop.hbase.testclassification.LargeTests")
public class TestImportTsv
extends Object
implements org.apache.hadoop.conf.Configurable

Field Summary

Fields

Modifier and Type	Field	Description
private Map<String,String>	args
protected static final String	DELETE_AFTER_LOAD_CONF
private final String	FAMILY
protected static final String	FORCE_COMBINER_CONF	Force use of combiner in doMROnTableTest.
private static final org.slf4j.Logger	LOG
protected static final String	NAME
private org.apache.hadoop.hbase.TableName	tn
protected static HBaseTestingUtil	util

Constructor Summary

Constructors

Constructor	Description
TestImportTsv()

Method Summary

Modifier and Type	Method	Description
private org.apache.hadoop.util.Tool	doMROnTableTest(String data, int valueMultiplier)
protected static org.apache.hadoop.util.Tool	doMROnTableTest(HBaseTestingUtil util, org.apache.hadoop.hbase.TableName table, String family, String data, Map<String,String> args)
protected static org.apache.hadoop.util.Tool	doMROnTableTest(HBaseTestingUtil util, org.apache.hadoop.hbase.TableName table, String family, String data, Map<String,String> args, int valueMultiplier, int expectedKVCount)	Run an ImportTsv job and perform basic validation on the results.
org.apache.hadoop.conf.Configuration	getConf()
private static int	getKVCountFromHfile(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path p)	Method returns the total KVs in given hfile
static void	provisionCluster()
static void	releaseCluster()
void	setConf(org.apache.hadoop.conf.Configuration conf)
void	setup()
void	testBulkOutputWithAnExistingTable()
void	testBulkOutputWithAnExistingTableNoStrictTrue()
void	testBulkOutputWithoutAnExistingTable()
void	testBulkOutputWithTsvImporterTextMapper()
void	testDryModeWithBulkModeAndTableDoesNotExistsCreateTableSetToYes()
void	testDryModeWithBulkOutputAndTableDoesNotExistsCreateTableSetToNo()	If table is not present in bulk mode and create.table is not set to yes, import should fail with TableNotFoundException.
void	testDryModeWithBulkOutputAndTableExists()
void	testDryModeWithoutBulkOutputAndTableDoesNotExists()	If table is not present in non-bulk mode, dry run should fail just like normal mode.
void	testDryModeWithoutBulkOutputAndTableExists()
void	testJobConfigurationsWithDryMode()
void	testJobConfigurationsWithTsvImporterTextMapper()
void	testMRNoMatchedColumnFamily()
void	testMROnTable()
void	testMROnTableWithCustomMapper()
void	testMROnTableWithTimestamp()
void	testMRWithoutAnExistingTable()
void	testSkipEmptyColumns()
void	testTsvImporterTextMapperWithInvalidData()	If there are invalid data rows as inputs, then only those rows should be ignored.
void	testWithoutAnExistingTableAndCreateTableSetToNo()
private static void	validateHFiles(org.apache.hadoop.fs.FileSystem fs, String outputPath, String family, int expectedKVCount)	Confirm ImportTsv via HFiles on fs.
private static void	validateTable(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.hbase.TableName tableName, String family, int valueMultiplier, boolean isDryRun)	Confirm ImportTsv via data in online table.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

LOG

private static final org.slf4j.Logger LOG

NAME

protected static final String NAME

util

protected static HBaseTestingUtil util

DELETE_AFTER_LOAD_CONF

protected static final String DELETE_AFTER_LOAD_CONF

FORCE_COMBINER_CONF

protected static final String FORCE_COMBINER_CONF

Force use of combiner in doMROnTableTest. Boolean. Default is true.

FAMILY

private final String FAMILY

See Also:

• Constant Field Values

tn

private org.apache.hadoop.hbase.TableName tn

args

private Map<String,String> args

Constructor Details

TestImportTsv

public TestImportTsv()

Method Details

getConf

public org.apache.hadoop.conf.Configuration getConf()

Specified by:

getConf in interface org.apache.hadoop.conf.Configurable

setConf

public void setConf(org.apache.hadoop.conf.Configuration conf)

Specified by:

setConf in interface org.apache.hadoop.conf.Configurable

provisionCluster

@BeforeAll
public static void provisionCluster()
                             throws Exception

Throws:

Exception

releaseCluster

@AfterAll
public static void releaseCluster()
                           throws Exception

Throws:

Exception

setup

@BeforeEach
public void setup()
           throws Exception

Throws:

Exception

testMROnTable

@Test
public void testMROnTable()
                   throws Exception

Throws:

Exception

testMROnTableWithTimestamp

@Test
public void testMROnTableWithTimestamp()
                                throws Exception

Throws:

Exception

testMROnTableWithCustomMapper

@Test
public void testMROnTableWithCustomMapper()
                                   throws Exception

Throws:

Exception

testBulkOutputWithoutAnExistingTable

@Test
public void testBulkOutputWithoutAnExistingTable()
                                          throws Exception

Throws:

Exception

testBulkOutputWithAnExistingTable

@Test
public void testBulkOutputWithAnExistingTable()
                                       throws Exception

Throws:

Exception

testBulkOutputWithAnExistingTableNoStrictTrue

@Test
public void testBulkOutputWithAnExistingTableNoStrictTrue()
                                                   throws Exception

Throws:

Exception

testJobConfigurationsWithTsvImporterTextMapper

@Test
public void testJobConfigurationsWithTsvImporterTextMapper()
                                                    throws Exception

Throws:

Exception

testBulkOutputWithTsvImporterTextMapper

@Test
public void testBulkOutputWithTsvImporterTextMapper()
                                             throws Exception

Throws:

Exception

testWithoutAnExistingTableAndCreateTableSetToNo

@Test
public void testWithoutAnExistingTableAndCreateTableSetToNo()
                                                     throws Exception

Throws:

Exception

testMRNoMatchedColumnFamily

@Test
public void testMRNoMatchedColumnFamily()
                                 throws Exception

Throws:

Exception

testMRWithoutAnExistingTable

@Test
public void testMRWithoutAnExistingTable()
                                  throws Exception

Throws:

Exception

testJobConfigurationsWithDryMode

@Test
public void testJobConfigurationsWithDryMode()
                                      throws Exception

Throws:

Exception

testDryModeWithoutBulkOutputAndTableExists

@Test
public void testDryModeWithoutBulkOutputAndTableExists()
                                                throws Exception

Throws:

Exception

testDryModeWithoutBulkOutputAndTableDoesNotExists

@Test
public void testDryModeWithoutBulkOutputAndTableDoesNotExists()
                                                       throws Exception

If table is not present in non-bulk mode, dry run should fail just like normal mode.

Throws:

Exception

testDryModeWithBulkOutputAndTableExists

@Test
public void testDryModeWithBulkOutputAndTableExists()
                                             throws Exception

Throws:

Exception

testDryModeWithBulkOutputAndTableDoesNotExistsCreateTableSetToNo

@Test
public void testDryModeWithBulkOutputAndTableDoesNotExistsCreateTableSetToNo()
                                                                      throws Exception

If table is not present in bulk mode and create.table is not set to yes, import should fail with TableNotFoundException.

Throws:

Exception

testDryModeWithBulkModeAndTableDoesNotExistsCreateTableSetToYes

@Test
public void testDryModeWithBulkModeAndTableDoesNotExistsCreateTableSetToYes()
                                                                     throws Exception

Throws:

Exception

testTsvImporterTextMapperWithInvalidData

@Test
public void testTsvImporterTextMapperWithInvalidData()
                                              throws Exception

If there are invalid data rows as inputs, then only those rows should be ignored.

Throws:

Exception

testSkipEmptyColumns

@Test
public void testSkipEmptyColumns()
                          throws Exception

Throws:

Exception

doMROnTableTest

private org.apache.hadoop.util.Tool doMROnTableTest(String data,
 int valueMultiplier)
                                             throws Exception

Throws:

Exception

doMROnTableTest

protected static org.apache.hadoop.util.Tool doMROnTableTest(HBaseTestingUtil util,
 org.apache.hadoop.hbase.TableName table,
 String family,
 String data,
 Map<String,String> args)
                                                      throws Exception

Throws:

Exception

doMROnTableTest

protected static org.apache.hadoop.util.Tool doMROnTableTest(HBaseTestingUtil util,
 org.apache.hadoop.hbase.TableName table,
 String family,
 String data,
 Map<String,String> args,
 int valueMultiplier,
 int expectedKVCount)
                                                      throws Exception

Run an ImportTsv job and perform basic validation on the results. Returns the ImportTsv Tool instance so that other tests can inspect it for further validation as necessary. This method is static to insure non-reliance on instance's util/conf facilities.

Parameters:

args - Any arguments to pass BEFORE inputFile path is appended.

Returns:

The Tool instance used to run the test.

Throws:

Exception

validateTable

private static void validateTable(org.apache.hadoop.conf.Configuration conf,
 org.apache.hadoop.hbase.TableName tableName,
 String family,
 int valueMultiplier,
 boolean isDryRun)
                           throws IOException

Confirm ImportTsv via data in online table.

Throws:

IOException

validateHFiles

private static void validateHFiles(org.apache.hadoop.fs.FileSystem fs,
 String outputPath,
 String family,
 int expectedKVCount)
                            throws IOException

Confirm ImportTsv via HFiles on fs.

Throws:

IOException

getKVCountFromHfile

private static int getKVCountFromHfile(org.apache.hadoop.fs.FileSystem fs,
 org.apache.hadoop.fs.Path p)
                                throws IOException

Method returns the total KVs in given hfile

Parameters:

fs - File System

p - HFile path

Returns:

KV count in the given hfile

Throws:

IOException