org.apache.hadoop.hbase.test

Class IntegrationTestLoadAndVerify

java.lang.Object

org.apache.hadoop.hbase.util.AbstractHBaseTool

org.apache.hadoop.hbase.IntegrationTestBase

org.apache.hadoop.hbase.test.IntegrationTestLoadAndVerify

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, org.apache.hadoop.util.Tool

Direct Known Subclasses:

IntegrationTestWithCellVisibilityLoadAndVerify

public class IntegrationTestLoadAndVerify
extends IntegrationTestBase

A large test which loads a lot of data that has internal references, and verifies the data. In load step, 200 map tasks are launched, which in turn write loadmapper.num_to_write (default 100K) rows to an hbase table. Rows are written in blocks, for a total of 100 blocks. Each row in a block, contains loadmapper.backrefs (default 50) references to random rows in the prev block. Verify step is scans the table, and verifies that for every referenced row, the row is actually there (no data loss). Failed rows are output from reduce to be saved in the job output dir in hdfs and inspected later. This class can be run as a unit test, as an integration test, or from the command line Originally taken from Apache Bigtop.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
private static class	IntegrationTestLoadAndVerify.Counters
static class	IntegrationTestLoadAndVerify.LoadMapper
static class	IntegrationTestLoadAndVerify.VerifyMapper
static class	IntegrationTestLoadAndVerify.VerifyReducer
static class	IntegrationTestLoadAndVerify.WALSearcher Tool to search missing rows in WALs and hfiles.

Field Summary

Fields

Modifier and Type	Field and Description
(package private) static String	FOUND_GROUP_KEY
private String	keysDir
private static org.slf4j.Logger	LOG
private static int	MISSING_ROWS_TO_LOG
private static int	NUM_BACKREFS_DEFAULT
private static String	NUM_BACKREFS_KEY
private static int	NUM_MAP_TASKS_DEFAULT
private static String	NUM_MAP_TASKS_KEY
private static int	NUM_REDUCE_TASKS_DEFAULT
private static String	NUM_REDUCE_TASKS_KEY
private static long	NUM_TO_WRITE_DEFAULT
private static String	NUM_TO_WRITE_KEY
private static int	SCANNER_CACHING
(package private) static String	SEARCHER_INPUTDIR_KEY
private static String	TABLE_NAME_DEFAULT
private static String	TABLE_NAME_KEY
private static byte[]	TEST_FAMILY
private static String	TEST_NAME
private static byte[]	TEST_QUALIFIER
private String	toRun

Fields inherited from class org.apache.hadoop.hbase.IntegrationTestBase

CHAOS_MONKEY_PROPS, monkey, MONKEY_LONG_OPT, monkeyProps, monkeyToUse, NO_CLUSTER_CLEANUP_LONG_OPT, noClusterCleanUp, util

Fields inherited from class org.apache.hadoop.hbase.util.AbstractHBaseTool

cmdLineArgs, conf, EXIT_FAILURE, EXIT_SUCCESS, LONG_HELP_OPTION, options, SHORT_HELP_OPTION

Constructor Summary

Constructors

Constructor and Description
IntegrationTestLoadAndVerify()

Method Summary

Modifier and Type	Method and Description
void	cleanUpCluster()
protected org.apache.hadoop.mapreduce.Job	doLoad(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.hbase.HTableDescriptor htd)
private int	doSearch(org.apache.hadoop.conf.Configuration conf, String keysDir)
protected void	doVerify(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.hbase.HTableDescriptor htd)
protected Set<String>	getColumnFamilies() Provides the name of the CFs that are protected from random Chaos monkey activity (alter)
org.apache.hadoop.hbase.TableName	getTablename() Provides the name of the table that is protected from random Chaos monkey activity
org.apache.hadoop.fs.Path	getTestDir(String testName, String subdir)
static void	main(String[] argv)
void	printUsage()
protected void	processOptions(org.apache.hbase.thirdparty.org.apache.commons.cli.CommandLine cmd)
private static SortedSet<byte[]>	readFileToSearch(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.FileStatus keyFileStatus, SortedSet<byte[]> result)
(package private) static SortedSet<byte[]>	readKeysToSearch(org.apache.hadoop.conf.Configuration conf)
int	runTestFromCommandLine()
private static void	setJobScannerConf(org.apache.hadoop.mapreduce.Job job)
protected void	setMapperClass(org.apache.hadoop.mapreduce.Job job)
void	setUpCluster()
static long	swapLong(long value) Converts a "long" value between endian systems.
void	testLoadAndVerify()

Methods inherited from class org.apache.hadoop.hbase.IntegrationTestBase

addOptions, cleanUp, cleanUpMonkey, cleanUpMonkey, doWork, getConf, getDefaultMonkeyFactory, getTestingUtil, processBaseOptions, setUp, setUpMonkey, startMonkey

Methods inherited from class org.apache.hadoop.hbase.util.AbstractHBaseTool

addOption, addOptNoArg, addOptNoArg, addOptWithArg, addOptWithArg, addRequiredOption, addRequiredOptWithArg, addRequiredOptWithArg, doStaticMain, getOptionAsDouble, getOptionAsInt, getOptionAsLong, newParser, parseArgs, parseInt, parseLong, printUsage, processOldArgs, run, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

private static final org.slf4j.Logger LOG

TEST_NAME

private static final String TEST_NAME

See Also:

Constant Field Values

TEST_FAMILY

private static final byte[] TEST_FAMILY

TEST_QUALIFIER

private static final byte[] TEST_QUALIFIER

NUM_TO_WRITE_KEY

private static final String NUM_TO_WRITE_KEY

See Also:

Constant Field Values

NUM_TO_WRITE_DEFAULT

private static final long NUM_TO_WRITE_DEFAULT

See Also:

Constant Field Values

TABLE_NAME_KEY

private static final String TABLE_NAME_KEY

See Also:

Constant Field Values

TABLE_NAME_DEFAULT

private static final String TABLE_NAME_DEFAULT

See Also:

Constant Field Values

NUM_BACKREFS_KEY

private static final String NUM_BACKREFS_KEY

See Also:

Constant Field Values

NUM_BACKREFS_DEFAULT

private static final int NUM_BACKREFS_DEFAULT

See Also:

Constant Field Values

NUM_MAP_TASKS_KEY

private static final String NUM_MAP_TASKS_KEY

See Also:

Constant Field Values

NUM_REDUCE_TASKS_KEY

private static final String NUM_REDUCE_TASKS_KEY

See Also:

Constant Field Values

NUM_MAP_TASKS_DEFAULT

private static final int NUM_MAP_TASKS_DEFAULT

See Also:

Constant Field Values

NUM_REDUCE_TASKS_DEFAULT

private static final int NUM_REDUCE_TASKS_DEFAULT

See Also:

Constant Field Values

SCANNER_CACHING

private static final int SCANNER_CACHING

See Also:

Constant Field Values

MISSING_ROWS_TO_LOG

private static final int MISSING_ROWS_TO_LOG

See Also:

Constant Field Values

toRun

private String toRun

keysDir

private String keysDir

FOUND_GROUP_KEY

static final String FOUND_GROUP_KEY

See Also:

Constant Field Values

SEARCHER_INPUTDIR_KEY

static final String SEARCHER_INPUTDIR_KEY

See Also:

Constant Field Values

Constructor Detail

IntegrationTestLoadAndVerify

public IntegrationTestLoadAndVerify()

Method Detail

setUpCluster

public void setUpCluster()
                  throws Exception

Specified by:

setUpCluster in class IntegrationTestBase

Throws:

Exception

cleanUpCluster

public void cleanUpCluster()
                    throws Exception

Overrides:

cleanUpCluster in class IntegrationTestBase

Throws:

Exception

swapLong

public static long swapLong(long value)

Converts a "long" value between endian systems. Borrowed from Apache Commons IO

Parameters:

value - value to convert

Returns:

the converted value

doLoad

protected org.apache.hadoop.mapreduce.Job doLoad(org.apache.hadoop.conf.Configuration conf,
                                                 org.apache.hadoop.hbase.HTableDescriptor htd)
                                          throws Exception

Throws:

Exception

setMapperClass

protected void setMapperClass(org.apache.hadoop.mapreduce.Job job)

doVerify

protected void doVerify(org.apache.hadoop.conf.Configuration conf,
                        org.apache.hadoop.hbase.HTableDescriptor htd)
                 throws Exception

Throws:

Exception

readKeysToSearch

static SortedSet<byte[]> readKeysToSearch(org.apache.hadoop.conf.Configuration conf)
                                   throws IOException,
                                          InterruptedException

Throws:

IOException

InterruptedException

readFileToSearch

private static SortedSet<byte[]> readFileToSearch(org.apache.hadoop.conf.Configuration conf,
                                                  org.apache.hadoop.fs.FileSystem fs,
                                                  org.apache.hadoop.fs.FileStatus keyFileStatus,
                                                  SortedSet<byte[]> result)
                                           throws IOException,
                                                  InterruptedException

Throws:

IOException

InterruptedException

doSearch

private int doSearch(org.apache.hadoop.conf.Configuration conf,
                     String keysDir)
              throws Exception

Throws:

Exception

setJobScannerConf

private static void setJobScannerConf(org.apache.hadoop.mapreduce.Job job)

getTestDir

public org.apache.hadoop.fs.Path getTestDir(String testName,
                                            String subdir)
                                     throws IOException

Throws:

IOException

testLoadAndVerify

public void testLoadAndVerify()
                       throws Exception

Throws:

Exception

printUsage

public void printUsage()

Overrides:

printUsage in class org.apache.hadoop.hbase.util.AbstractHBaseTool

processOptions

protected void processOptions(org.apache.hbase.thirdparty.org.apache.commons.cli.CommandLine cmd)

Overrides:

processOptions in class IntegrationTestBase

runTestFromCommandLine

public int runTestFromCommandLine()
                           throws Exception

Specified by:

runTestFromCommandLine in class IntegrationTestBase

Throws:

Exception

getTablename

public org.apache.hadoop.hbase.TableName getTablename()

Description copied from class: IntegrationTestBase

Provides the name of the table that is protected from random Chaos monkey activity

Specified by:

getTablename in class IntegrationTestBase

Returns:

table to not delete.

getColumnFamilies

protected Set<String> getColumnFamilies()

Description copied from class: IntegrationTestBase

Provides the name of the CFs that are protected from random Chaos monkey activity (alter)

Specified by:

getColumnFamilies in class IntegrationTestBase

Returns:

set of cf names to protect.

main

public static void main(String[] argv)
                 throws Exception

Throws:

Exception