org.apache.hadoop.hbase.mapreduce

Class TestImportTsv

java.lang.Object

org.apache.hadoop.hbase.mapreduce.TestImportTsv

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable

public class TestImportTsv
extends Object
implements org.apache.hadoop.conf.Configurable

Field Summary

Fields

Modifier and Type	Field and Description
private Map<String,String>	args
static HBaseClassTestRule	CLASS_RULE
protected static String	DELETE_AFTER_LOAD_CONF
org.junit.rules.ExpectedException	exception
private String	FAMILY
protected static String	FORCE_COMBINER_CONF Force use of combiner in doMROnTableTest.
private static org.slf4j.Logger	LOG
protected static String	NAME
private org.apache.hadoop.hbase.TableName	tn
protected static HBaseTestingUtility	util

Constructor Summary

Constructors

Constructor and Description
TestImportTsv()

Method Summary

Modifier and Type	Method and Description
protected static org.apache.hadoop.util.Tool	doMROnTableTest(HBaseTestingUtility util, org.apache.hadoop.hbase.TableName table, String family, String data, Map<String,String> args)
protected static org.apache.hadoop.util.Tool	doMROnTableTest(HBaseTestingUtility util, org.apache.hadoop.hbase.TableName table, String family, String data, Map<String,String> args, int valueMultiplier, int expectedKVCount) Run an ImportTsv job and perform basic validation on the results.
private org.apache.hadoop.util.Tool	doMROnTableTest(String data, int valueMultiplier)
org.apache.hadoop.conf.Configuration	getConf()
private static int	getKVCountFromHfile(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path p) Method returns the total KVs in given hfile
static void	provisionCluster()
static void	releaseCluster()
void	setConf(org.apache.hadoop.conf.Configuration conf)
void	setup()
void	testBulkOutputWithAnExistingTable()
void	testBulkOutputWithAnExistingTableNoStrictTrue()
void	testBulkOutputWithoutAnExistingTable()
void	testBulkOutputWithTsvImporterTextMapper()
void	testDryModeWithBulkModeAndTableDoesNotExistsCreateTableSetToYes()
void	testDryModeWithBulkOutputAndTableDoesNotExistsCreateTableSetToNo() If table is not present in bulk mode and create.table is not set to yes, import should fail with TableNotFoundException.
void	testDryModeWithBulkOutputAndTableExists()
void	testDryModeWithoutBulkOutputAndTableDoesNotExists() If table is not present in non-bulk mode, dry run should fail just like normal mode.
void	testDryModeWithoutBulkOutputAndTableExists()
void	testJobConfigurationsWithDryMode()
void	testJobConfigurationsWithTsvImporterTextMapper()
void	testMROnTable()
void	testMROnTableWithCustomMapper()
void	testMROnTableWithTimestamp()
void	testMRWithoutAnExistingTable()
void	testSkipEmptyColumns()
void	testTsvImporterTextMapperWithInvalidData() If there are invalid data rows as inputs, then only those rows should be ignored.
void	testWithoutAnExistingTableAndCreateTableSetToNo()
private static void	validateHFiles(org.apache.hadoop.fs.FileSystem fs, String outputPath, String family, int expectedKVCount) Confirm ImportTsv via HFiles on fs.
private static void	validateTable(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.hbase.TableName tableName, String family, int valueMultiplier, boolean isDryRun) Confirm ImportTsv via data in online table.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

CLASS_RULE

public static final HBaseClassTestRule CLASS_RULE

LOG

private static final org.slf4j.Logger LOG

NAME

protected static final String NAME

util

protected static HBaseTestingUtility util

DELETE_AFTER_LOAD_CONF

protected static final String DELETE_AFTER_LOAD_CONF

FORCE_COMBINER_CONF

protected static final String FORCE_COMBINER_CONF

Force use of combiner in doMROnTableTest. Boolean. Default is true.

FAMILY

private final String FAMILY

See Also:

Constant Field Values

tn

private org.apache.hadoop.hbase.TableName tn

args

private Map<String,String> args

exception

public org.junit.rules.ExpectedException exception

Constructor Detail

TestImportTsv

public TestImportTsv()

Method Detail

getConf

public org.apache.hadoop.conf.Configuration getConf()

Specified by:

getConf in interface org.apache.hadoop.conf.Configurable

setConf

public void setConf(org.apache.hadoop.conf.Configuration conf)

Specified by:

setConf in interface org.apache.hadoop.conf.Configurable

provisionCluster

public static void provisionCluster()
                             throws Exception

Throws:

Exception

releaseCluster

public static void releaseCluster()
                           throws Exception

Throws:

Exception

setup

public void setup()
           throws Exception

Throws:

Exception

testMROnTable

public void testMROnTable()
                   throws Exception

Throws:

Exception

testMROnTableWithTimestamp

public void testMROnTableWithTimestamp()
                                throws Exception

Throws:

Exception

testMROnTableWithCustomMapper

public void testMROnTableWithCustomMapper()
                                   throws Exception

Throws:

Exception

testBulkOutputWithoutAnExistingTable

public void testBulkOutputWithoutAnExistingTable()
                                          throws Exception

Throws:

Exception

testBulkOutputWithAnExistingTable

public void testBulkOutputWithAnExistingTable()
                                       throws Exception

Throws:

Exception

testBulkOutputWithAnExistingTableNoStrictTrue

public void testBulkOutputWithAnExistingTableNoStrictTrue()
                                                   throws Exception

Throws:

Exception

testJobConfigurationsWithTsvImporterTextMapper

public void testJobConfigurationsWithTsvImporterTextMapper()
                                                    throws Exception

Throws:

Exception

testBulkOutputWithTsvImporterTextMapper

public void testBulkOutputWithTsvImporterTextMapper()
                                             throws Exception

Throws:

Exception

testWithoutAnExistingTableAndCreateTableSetToNo

public void testWithoutAnExistingTableAndCreateTableSetToNo()
                                                     throws Exception

Throws:

Exception

testMRWithoutAnExistingTable

public void testMRWithoutAnExistingTable()
                                  throws Exception

Throws:

Exception

testJobConfigurationsWithDryMode

public void testJobConfigurationsWithDryMode()
                                      throws Exception

Throws:

Exception

testDryModeWithoutBulkOutputAndTableExists

public void testDryModeWithoutBulkOutputAndTableExists()
                                                throws Exception

Throws:

Exception

testDryModeWithoutBulkOutputAndTableDoesNotExists

public void testDryModeWithoutBulkOutputAndTableDoesNotExists()
                                                       throws Exception

If table is not present in non-bulk mode, dry run should fail just like normal mode.

Throws:

Exception

testDryModeWithBulkOutputAndTableExists

public void testDryModeWithBulkOutputAndTableExists()
                                             throws Exception

Throws:

Exception

testDryModeWithBulkOutputAndTableDoesNotExistsCreateTableSetToNo

public void testDryModeWithBulkOutputAndTableDoesNotExistsCreateTableSetToNo()
                                                                      throws Exception

If table is not present in bulk mode and create.table is not set to yes, import should fail with TableNotFoundException.

Throws:

Exception

testDryModeWithBulkModeAndTableDoesNotExistsCreateTableSetToYes

public void testDryModeWithBulkModeAndTableDoesNotExistsCreateTableSetToYes()
                                                                     throws Exception

Throws:

Exception

testTsvImporterTextMapperWithInvalidData

public void testTsvImporterTextMapperWithInvalidData()
                                              throws Exception

If there are invalid data rows as inputs, then only those rows should be ignored.

Throws:

Exception

testSkipEmptyColumns

public void testSkipEmptyColumns()
                          throws Exception

Throws:

Exception

doMROnTableTest

private org.apache.hadoop.util.Tool doMROnTableTest(String data,
                                                    int valueMultiplier)
                                             throws Exception

Throws:

Exception

doMROnTableTest

protected static org.apache.hadoop.util.Tool doMROnTableTest(HBaseTestingUtility util,
                                                             org.apache.hadoop.hbase.TableName table,
                                                             String family,
                                                             String data,
                                                             Map<String,String> args)
                                                      throws Exception

Throws:

Exception

doMROnTableTest

protected static org.apache.hadoop.util.Tool doMROnTableTest(HBaseTestingUtility util,
                                                             org.apache.hadoop.hbase.TableName table,
                                                             String family,
                                                             String data,
                                                             Map<String,String> args,
                                                             int valueMultiplier,
                                                             int expectedKVCount)
                                                      throws Exception

Run an ImportTsv job and perform basic validation on the results. Returns the ImportTsv Tool instance so that other tests can inspect it for further validation as necessary. This method is static to insure non-reliance on instance's util/conf facilities.

Parameters:

args - Any arguments to pass BEFORE inputFile path is appended.

Returns:

The Tool instance used to run the test.

Throws:

Exception

validateTable

private static void validateTable(org.apache.hadoop.conf.Configuration conf,
                                  org.apache.hadoop.hbase.TableName tableName,
                                  String family,
                                  int valueMultiplier,
                                  boolean isDryRun)
                           throws IOException

Confirm ImportTsv via data in online table.

Throws:

IOException

validateHFiles

private static void validateHFiles(org.apache.hadoop.fs.FileSystem fs,
                                   String outputPath,
                                   String family,
                                   int expectedKVCount)
                            throws IOException

Confirm ImportTsv via HFiles on fs.

Throws:

IOException

getKVCountFromHfile

private static int getKVCountFromHfile(org.apache.hadoop.fs.FileSystem fs,
                                       org.apache.hadoop.fs.Path p)
                                throws IOException

Method returns the total KVs in given hfile

Parameters:

fs - File System

p - HFile path

Returns:

KV count in the given hfile

Throws:

IOException