org.apache.hadoop.hbase.util

Class MultiThreadedAction

java.lang.Object

org.apache.hadoop.hbase.util.MultiThreadedAction

Direct Known Subclasses:

MultiThreadedReader, MultiThreadedWriterBase

public abstract class MultiThreadedAction
extends Object

Common base class for reader and writer parts of multi-thread HBase load test (See LoadTestTool).

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	MultiThreadedAction.DefaultDataGenerator Default implementation of LoadTestDataGenerator that uses LoadTestKVGenerator, fixed set of column families, and random number of columns in range.
private class	MultiThreadedAction.ProgressReporter Asynchronously reports progress

Field Summary

Fields

Modifier and Type	Field and Description
private String	actionLetter "R" or "W"
protected org.apache.hadoop.conf.Configuration	conf
protected org.apache.hadoop.hbase.client.ClusterConnection	connection
protected LoadTestDataGenerator	dataGenerator
protected long	endKey The end key of the key range, exclusive
private static org.slf4j.Logger	LOG
protected AtomicLong	numCols
protected AtomicLong	numKeys
protected int	numThreads
protected AtomicInteger	numThreadsWorking
static int	REPORTING_INTERVAL_MS
protected long	startKey The start key of the key range, inclusive
private boolean	streamingCounters Whether we need to print out Hadoop Streaming-style counters
protected org.apache.hadoop.hbase.TableName	tableName
protected AtomicLong	totalOpTimeMs
protected boolean	verbose

Constructor Summary

Constructors

Constructor and Description
MultiThreadedAction(LoadTestDataGenerator dataGen, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.hbase.TableName tableName, String actionLetter)

Method Summary

Modifier and Type	Method and Description
protected static void	appendToStatus(StringBuilder sb, String desc, long v)
protected static void	appendToStatus(StringBuilder sb, String desc, String v)
void	close()
private static String	formatTime(long elapsedTime)
long	getEndKey() Returns the end key of the key range, exclusive
boolean	isDone()
private Map<String,org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos.MutationProto.MutationType>	parseMutateInfo(byte[] mutateInfo)
private void	printLocations(org.apache.hadoop.hbase.client.Result r)
protected abstract String	progressInfo() Returns a task-specific progress string
private String	resultToString(org.apache.hadoop.hbase.client.Result result)
void	start(long startKey, long endKey, int numThreads)
protected void	startThreads(Collection<? extends Thread> threads)
boolean	verifyResultAgainstDataGenerator(org.apache.hadoop.hbase.client.Result result, boolean verifyValues) See verifyResultAgainstDataGenerator(Result, boolean, boolean).
boolean	verifyResultAgainstDataGenerator(org.apache.hadoop.hbase.client.Result result, boolean verifyValues, boolean verifyCfAndColumnIntegrity) Verifies the result from get or scan using the dataGenerator (that was presumably also used to generate said result).
void	waitForFinish()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

private static final org.slf4j.Logger LOG

tableName

protected final org.apache.hadoop.hbase.TableName tableName

conf

protected final org.apache.hadoop.conf.Configuration conf

connection

protected final org.apache.hadoop.hbase.client.ClusterConnection connection

numThreads

protected int numThreads

startKey

protected long startKey

The start key of the key range, inclusive

endKey

protected long endKey

The end key of the key range, exclusive

numThreadsWorking

protected AtomicInteger numThreadsWorking

numKeys

protected AtomicLong numKeys

numCols

protected AtomicLong numCols

totalOpTimeMs

protected AtomicLong totalOpTimeMs

verbose

protected boolean verbose

dataGenerator

protected LoadTestDataGenerator dataGenerator

actionLetter

private String actionLetter

"R" or "W"

streamingCounters

private boolean streamingCounters

Whether we need to print out Hadoop Streaming-style counters

REPORTING_INTERVAL_MS

public static final int REPORTING_INTERVAL_MS

See Also:

Constant Field Values

Constructor Detail

MultiThreadedAction

public MultiThreadedAction(LoadTestDataGenerator dataGen,
                           org.apache.hadoop.conf.Configuration conf,
                           org.apache.hadoop.hbase.TableName tableName,
                           String actionLetter)
                    throws IOException

Throws:

IOException

Method Detail

start

public void start(long startKey,
                  long endKey,
                  int numThreads)
           throws IOException

Throws:

IOException

formatTime

private static String formatTime(long elapsedTime)

close

public void close()

waitForFinish

public void waitForFinish()

isDone

public boolean isDone()

startThreads

protected void startThreads(Collection<? extends Thread> threads)

getEndKey

public long getEndKey()

Returns the end key of the key range, exclusive

progressInfo

protected abstract String progressInfo()

Returns a task-specific progress string

appendToStatus

protected static void appendToStatus(StringBuilder sb,
                                     String desc,
                                     long v)

appendToStatus

protected static void appendToStatus(StringBuilder sb,
                                     String desc,
                                     String v)

verifyResultAgainstDataGenerator

public boolean verifyResultAgainstDataGenerator(org.apache.hadoop.hbase.client.Result result,
                                                boolean verifyValues)

See verifyResultAgainstDataGenerator(Result, boolean, boolean). Does not verify cf/column integrity.

verifyResultAgainstDataGenerator

public boolean verifyResultAgainstDataGenerator(org.apache.hadoop.hbase.client.Result result,
                                                boolean verifyValues,
                                                boolean verifyCfAndColumnIntegrity)

Verifies the result from get or scan using the dataGenerator (that was presumably also used to generate said result).

Parameters:

verifyValues - verify that values in the result make sense for row/cf/column combination

verifyCfAndColumnIntegrity - verify that cf/column set in the result is complete. Note that to use this multiPut should be used, or verification has to happen after writes, otherwise there can be races.

Returns:

true if the values of row result makes sense for row/cf/column combination and true if the cf/column set in the result is complete, false otherwise.

printLocations

private void printLocations(org.apache.hadoop.hbase.client.Result r)

resultToString

private String resultToString(org.apache.hadoop.hbase.client.Result result)

parseMutateInfo

private Map<String,org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos.MutationProto.MutationType> parseMutateInfo(byte[] mutateInfo)