org.apache.hadoop.hbase.mapreduce

Class IntegrationTestFileBasedSFTBulkLoad

java.lang.Object

org.apache.hadoop.hbase.util.AbstractHBaseTool

org.apache.hadoop.hbase.IntegrationTestBase

org.apache.hadoop.hbase.mapreduce.IntegrationTestBulkLoad

org.apache.hadoop.hbase.mapreduce.IntegrationTestFileBasedSFTBulkLoad

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, org.apache.hadoop.util.Tool

public class IntegrationTestFileBasedSFTBulkLoad
extends IntegrationTestBulkLoad

Test Bulk Load and MR on a distributed cluster. With FileBased StorefileTracker enabled. It starts an MR job that creates linked chains The format of rows is like this: Row Key -> Long L:<< Chain Id >> -> Row Key of the next link in the chain S:<< Chain Id >> -> The step in the chain that his link is. D:<< Chain Id >> -> Random Data. All chains start on row 0. All rk's are > 0. After creating the linked lists they are walked over using a TableMapper based Mapreduce Job. There are a few options exposed: hbase.IntegrationTestBulkLoad.chainLength The number of rows that will be part of each and every chain. hbase.IntegrationTestBulkLoad.numMaps The number of mappers that will be run. Each mapper creates on linked list chain. hbase.IntegrationTestBulkLoad.numImportRounds How many jobs will be run to create linked lists. hbase.IntegrationTestBulkLoad.tableName The name of the table. hbase.IntegrationTestBulkLoad.replicaCount How many region replicas to configure for the table under test.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.hbase.mapreduce.IntegrationTestBulkLoad

IntegrationTestBulkLoad.CompositeKeyComparator, IntegrationTestBulkLoad.EmptySplit, IntegrationTestBulkLoad.FixedRecordReader<K,V>, IntegrationTestBulkLoad.ITBulkLoadInputFormat, IntegrationTestBulkLoad.LinkChain, IntegrationTestBulkLoad.LinkedListCheckingMapper, IntegrationTestBulkLoad.LinkedListCheckingReducer, IntegrationTestBulkLoad.LinkedListCreationMapper, IntegrationTestBulkLoad.LinkKey, IntegrationTestBulkLoad.NaturalKeyGroupingComparator, IntegrationTestBulkLoad.NaturalKeyPartitioner, IntegrationTestBulkLoad.SlowMeCoproScanOperations

Nested classes/interfaces inherited from class org.apache.hadoop.hbase.util.AbstractHBaseTool

org.apache.hadoop.hbase.util.AbstractHBaseTool.OptionsOrderComparator

Field Summary

Fields

Modifier and Type	Field and Description
private static org.slf4j.Logger	LOG
private static String	NUM_IMPORT_ROUNDS_KEY
private static String	NUM_MAPS_KEY
private static int	NUM_REPLICA_COUNT_DEFAULT
private static String	NUM_REPLICA_COUNT_KEY

Fields inherited from class org.apache.hadoop.hbase.IntegrationTestBase

CHAOS_MONKEY_PROPS, monkey, MONKEY_LONG_OPT, monkeyProps, monkeyToUse, NO_CLUSTER_CLEANUP_LONG_OPT, noClusterCleanUp, util

Fields inherited from class org.apache.hadoop.hbase.util.AbstractHBaseTool

cmdLineArgs, conf, EXIT_FAILURE, EXIT_SUCCESS, LONG_HELP_OPTION, options, SHORT_HELP_OPTION

Constructor Summary

Constructors

Constructor and Description
IntegrationTestFileBasedSFTBulkLoad()

Method Summary

Modifier and Type	Method and Description
static void	main(String[] args)
void	setUpCluster()
void	testFileBasedSFTBulkLoad()

Methods inherited from class org.apache.hadoop.hbase.mapreduce.IntegrationTestBulkLoad

addOptions, getColumnFamilies, getTablename, getTableName, processOptions, runLoad, runTestFromCommandLine, testBulkLoad

Methods inherited from class org.apache.hadoop.hbase.IntegrationTestBase

cleanUp, cleanUpCluster, cleanUpMonkey, cleanUpMonkey, doWork, getConf, getDefaultMonkeyFactory, getTestingUtil, loadMonkeyProperties, processBaseOptions, setUp, setUpMonkey, startMonkey

Methods inherited from class org.apache.hadoop.hbase.util.AbstractHBaseTool

addOption, addOptNoArg, addOptNoArg, addOptWithArg, addOptWithArg, addRequiredOption, addRequiredOptWithArg, addRequiredOptWithArg, doStaticMain, getOptionAsDouble, getOptionAsInt, getOptionAsInt, getOptionAsLong, getOptionAsLong, newParser, parseArgs, parseInt, parseLong, printUsage, printUsage, processOldArgs, run, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

private static final org.slf4j.Logger LOG

NUM_MAPS_KEY

private static String NUM_MAPS_KEY

NUM_IMPORT_ROUNDS_KEY

private static String NUM_IMPORT_ROUNDS_KEY

NUM_REPLICA_COUNT_KEY

private static String NUM_REPLICA_COUNT_KEY

NUM_REPLICA_COUNT_DEFAULT

private static int NUM_REPLICA_COUNT_DEFAULT

Constructor Detail

IntegrationTestFileBasedSFTBulkLoad

public IntegrationTestFileBasedSFTBulkLoad()

Method Detail

testFileBasedSFTBulkLoad

public void testFileBasedSFTBulkLoad()
                              throws Exception

Throws:

Exception

setUpCluster

public void setUpCluster()
                  throws Exception

Overrides:

setUpCluster in class IntegrationTestBulkLoad

Throws:

Exception

main

public static void main(String[] args)
                 throws Exception

Throws:

Exception