org.apache.hadoop.hbase.mapreduce

Class IntegrationTestTableSnapshotInputFormat

java.lang.Object

org.apache.hadoop.hbase.util.AbstractHBaseTool

org.apache.hadoop.hbase.IntegrationTestBase

org.apache.hadoop.hbase.mapreduce.IntegrationTestTableSnapshotInputFormat

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, org.apache.hadoop.util.Tool

public class IntegrationTestTableSnapshotInputFormat
extends IntegrationTestBase

An integration test to test TableSnapshotInputFormat which enables reading directly from snapshot files without going through hbase servers. This test creates a table and loads the table with the rows ranging from 'aaa' to 'zzz', and for each row, sets the columns f1:(null) and f2:(null) to be the the same as the row value.

 aaa, f1: => aaa
 aaa, f2: => aaa
 aab, f1: => aab
 ....
 zzz, f2: => zzz
 

Then the test creates a snapshot from this table, and overrides the values in the original table with values 'after_snapshot_value'. The test, then runs a mapreduce job over the snapshot with a scan start row 'bbb' and stop row 'yyy'. The data is saved in a single reduce output file, and inspected later to verify that the MR job has seen all the values from the snapshot.

These parameters can be used to configure the job:
"IntegrationTestTableSnapshotInputFormat.table" => the name of the table
"IntegrationTestTableSnapshotInputFormat.snapshot" => the name of the snapshot
"IntegrationTestTableSnapshotInputFormat.numRegions" => number of regions in the table to be created (default, 32).
"IntegrationTestTableSnapshotInputFormat.tableDir" => temporary directory to restore the snapshot files

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.hbase.util.AbstractHBaseTool

org.apache.hadoop.hbase.util.AbstractHBaseTool.OptionsOrderComparator

Field Summary

Fields

Modifier and Type	Field and Description
private static int	DEFAULT_NUM_REGIONS
private static String	DEFAULT_TABLE_NAME
private static byte[]	END_ROW
private static org.slf4j.Logger	LOG
private static byte[]	MAPRED_END_ROW
private static String	MAPRED_IMPLEMENTATION
private static byte[]	MAPRED_START_ROW
private static String	MAPREDUCE_IMPLEMENTATION
private static String	MR_IMPLEMENTATION_KEY
private static String	NUM_REGIONS_KEY
private static String	SNAPSHOT_NAME_KEY
private static byte[]	START_ROW
private static String	TABLE_DIR_KEY
private static String	TABLE_NAME_KEY

Fields inherited from class org.apache.hadoop.hbase.IntegrationTestBase

CHAOS_MONKEY_PROPS, monkey, MONKEY_LONG_OPT, monkeyProps, monkeyToUse, NO_CLUSTER_CLEANUP_LONG_OPT, noClusterCleanUp, util

Fields inherited from class org.apache.hadoop.hbase.util.AbstractHBaseTool

cmdLineArgs, conf, EXIT_FAILURE, EXIT_SUCCESS, LONG_HELP_OPTION, options, SHORT_HELP_OPTION

Constructor Summary

Constructors

Constructor and Description
IntegrationTestTableSnapshotInputFormat()

Method Summary

Modifier and Type	Method and Description
void	cleanUp()
protected Set<String>	getColumnFamilies() Provides the name of the CFs that are protected from random Chaos monkey activity (alter)
org.apache.hadoop.hbase.TableName	getTablename() Provides the name of the table that is protected from random Chaos monkey activity
static void	main(String[] args)
int	runTestFromCommandLine()
void	setConf(org.apache.hadoop.conf.Configuration conf)
void	setUp()
void	setUpCluster()

Methods inherited from class org.apache.hadoop.hbase.IntegrationTestBase

addOptions, cleanUpCluster, cleanUpMonkey, cleanUpMonkey, doWork, getConf, getDefaultMonkeyFactory, getTestingUtil, loadMonkeyProperties, processBaseOptions, processOptions, setUpMonkey, startMonkey

Methods inherited from class org.apache.hadoop.hbase.util.AbstractHBaseTool

addOption, addOptNoArg, addOptNoArg, addOptWithArg, addOptWithArg, addRequiredOption, addRequiredOptWithArg, addRequiredOptWithArg, doStaticMain, getOptionAsDouble, getOptionAsInt, getOptionAsInt, getOptionAsLong, getOptionAsLong, newParser, parseArgs, parseInt, parseLong, printUsage, printUsage, processOldArgs, run

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

private static final org.slf4j.Logger LOG

TABLE_NAME_KEY

private static final String TABLE_NAME_KEY

See Also:

Constant Field Values

DEFAULT_TABLE_NAME

private static final String DEFAULT_TABLE_NAME

See Also:

Constant Field Values

SNAPSHOT_NAME_KEY

private static final String SNAPSHOT_NAME_KEY

See Also:

Constant Field Values

NUM_REGIONS_KEY

private static final String NUM_REGIONS_KEY

See Also:

Constant Field Values

MR_IMPLEMENTATION_KEY

private static final String MR_IMPLEMENTATION_KEY

See Also:

Constant Field Values

MAPRED_IMPLEMENTATION

private static final String MAPRED_IMPLEMENTATION

See Also:

Constant Field Values

MAPREDUCE_IMPLEMENTATION

private static final String MAPREDUCE_IMPLEMENTATION

See Also:

Constant Field Values

DEFAULT_NUM_REGIONS

private static final int DEFAULT_NUM_REGIONS

See Also:

Constant Field Values

TABLE_DIR_KEY

private static final String TABLE_DIR_KEY

See Also:

Constant Field Values

START_ROW

private static final byte[] START_ROW

END_ROW

private static final byte[] END_ROW

MAPRED_START_ROW

private static final byte[] MAPRED_START_ROW

MAPRED_END_ROW

private static final byte[] MAPRED_END_ROW

Constructor Detail

IntegrationTestTableSnapshotInputFormat

public IntegrationTestTableSnapshotInputFormat()

Method Detail

setConf

public void setConf(org.apache.hadoop.conf.Configuration conf)

Specified by:

setConf in interface org.apache.hadoop.conf.Configurable

Overrides:

setConf in class org.apache.hadoop.hbase.util.AbstractHBaseTool

setUp

public void setUp()
           throws Exception

Overrides:

setUp in class IntegrationTestBase

Throws:

Exception

cleanUp

public void cleanUp()
             throws Exception

Overrides:

cleanUp in class IntegrationTestBase

Throws:

Exception

setUpCluster

public void setUpCluster()
                  throws Exception

Specified by:

setUpCluster in class IntegrationTestBase

Throws:

Exception

runTestFromCommandLine

public int runTestFromCommandLine()
                           throws Exception

Specified by:

runTestFromCommandLine in class IntegrationTestBase

Throws:

Exception

getTablename

public org.apache.hadoop.hbase.TableName getTablename()

Description copied from class: IntegrationTestBase

Provides the name of the table that is protected from random Chaos monkey activity

Specified by:

getTablename in class IntegrationTestBase

Returns:

table to not delete.

getColumnFamilies

protected Set<String> getColumnFamilies()

Description copied from class: IntegrationTestBase

Provides the name of the CFs that are protected from random Chaos monkey activity (alter)

Specified by:

getColumnFamilies in class IntegrationTestBase

Returns:

set of cf names to protect.

main

public static void main(String[] args)
                 throws Exception

Throws:

Exception