Package org.apache.hadoop.hbase

Class PerformanceEvaluation

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.hbase.PerformanceEvaluation

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, org.apache.hadoop.util.Tool

@LimitedPrivate("Tools")
public class PerformanceEvaluation
extends org.apache.hadoop.conf.Configured
implements org.apache.hadoop.util.Tool

Script used evaluating HBase performance and scalability. Runs a HBase client that steps through one of a set of hardcoded tests or 'experiments' (e.g. a random reads test, a random writes test, etc.). Pass on the command-line which test to run and how many clients are participating in this experiment. Run PerformanceEvaluation --help to obtain usage.

This class sets up and runs the evaluation programs described in Section 7, Performance Evaluation, of the Bigtable paper, pages 8-10.

By default, runs as a mapreduce job where each mapper runs a single test client. Can also run as a non-mapreduce, multithreaded application by specifying --nomapred. Each client does about 1GB of data, unless specified otherwise.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
(package private) static class	PerformanceEvaluation.AppendTest
(package private) static class	PerformanceEvaluation.AsyncRandomReadTest
(package private) static class	PerformanceEvaluation.AsyncRandomWriteTest
(package private) static class	PerformanceEvaluation.AsyncScanTest
(package private) static class	PerformanceEvaluation.AsyncSequentialReadTest
(package private) static class	PerformanceEvaluation.AsyncSequentialWriteTest
(package private) static class	PerformanceEvaluation.AsyncTableTest
(package private) static class	PerformanceEvaluation.AsyncTest
(package private) static class	PerformanceEvaluation.BufferedMutatorTest
(package private) static class	PerformanceEvaluation.CASTableTest	Base class for operations that are CAS-like; that read a value and then set it based off what they read.
(package private) static class	PerformanceEvaluation.CheckAndDeleteTest
(package private) static class	PerformanceEvaluation.CheckAndMutateTest
(package private) static class	PerformanceEvaluation.CheckAndPutTest
(package private) static class	PerformanceEvaluation.CleanMetaTest
(package private) static class	PerformanceEvaluation.CmdDescriptor	Describes a command.
protected static enum	PerformanceEvaluation.Counter	Enum for map metrics.
static class	PerformanceEvaluation.EvaluationMapTask	MapReduce job that runs a performance evaluation client in each map task.
(package private) static class	PerformanceEvaluation.FilteredScanTest
(package private) static class	PerformanceEvaluation.IncrementTest
(package private) static class	PerformanceEvaluation.MetaRandomReadTest
(package private) static class	PerformanceEvaluation.MetaTest
(package private) static class	PerformanceEvaluation.MetaWriteTest
(package private) static class	PerformanceEvaluation.RandomDeleteTest
(package private) static class	PerformanceEvaluation.RandomReadTest
(package private) static class	PerformanceEvaluation.RandomScanWithRange10000Test
(package private) static class	PerformanceEvaluation.RandomScanWithRange1000Test
(package private) static class	PerformanceEvaluation.RandomScanWithRange100Test
(package private) static class	PerformanceEvaluation.RandomScanWithRange10Test
(package private) static class	PerformanceEvaluation.RandomScanWithRangeTest
(package private) static class	PerformanceEvaluation.RandomSeekScanTest
(package private) static class	PerformanceEvaluation.RandomWriteTest
(package private) static class	PerformanceEvaluation.ReverseScanTest
protected static class	PerformanceEvaluation.RunResult
(package private) static class	PerformanceEvaluation.ScanTest
(package private) static class	PerformanceEvaluation.SequentialDeleteTest
(package private) static class	PerformanceEvaluation.SequentialReadTest
(package private) static class	PerformanceEvaluation.SequentialWriteTest
(package private) static interface	PerformanceEvaluation.Status	Implementations can have their status set.
(package private) static class	PerformanceEvaluation.TableTest
(package private) static class	PerformanceEvaluation.Test
(package private) static class	PerformanceEvaluation.TestBase
(package private) static class	PerformanceEvaluation.TestOptions	Wraps up options passed to PerformanceEvaluation.

Field Summary

Fields

Modifier and Type	Field	Description
private static final BigDecimal	BYTES_PER_MB
static final byte[]	COLUMN_ZERO
private static Map<String,PerformanceEvaluation.CmdDescriptor>	COMMANDS
private static final MathContext	CXT
private static final PerformanceEvaluation.TestOptions	DEFAULT_OPTS
private static final int	DEFAULT_ROWS_PER_GB
static final int	DEFAULT_VALUE_LENGTH
static final String	FAMILY_NAME_BASE
static final byte[]	FAMILY_ZERO
private static final DecimalFormat	FMT
private static final org.apache.hbase.thirdparty.com.google.gson.Gson	GSON
(package private) static String	JOB_INPUT_FILENAME	Each client has one mapper to do the work, and client do the resulting count in a map task.
private static final org.slf4j.Logger	LOG
private static final BigDecimal	MS_PER_SEC
private static final int	ONE_GB
(package private) static final String	PE_COMMAND_SHORTNAME
private static final org.apache.hadoop.fs.Path	PERF_EVAL_DIR
(package private) static final String	RANDOM_READ
(package private) static final String	RANDOM_SEEK_SCAN
static final int	ROW_LENGTH
static final String	TABLE_NAME
private static final int	TAG_LENGTH

Constructor Summary

Constructors

Constructor	Description
PerformanceEvaluation(org.apache.hadoop.conf.Configuration conf)	Constructor

Method Summary

Modifier and Type	Method	Description
protected static void	addCommandDescriptor(Class<? extends PerformanceEvaluation.TestBase> cmdClass, String name, String description)
private static String	calculateMbps(int rows, long timeMs, int valueSize, int families, int columns)	Compute a throughput rate in MB/s.
(package private) static PerformanceEvaluation.TestOptions	calculateRowsAndSize(PerformanceEvaluation.TestOptions opts)
(package private) static boolean	checkTable(org.apache.hadoop.hbase.client.Admin admin, PerformanceEvaluation.TestOptions opts)
private static Class<? extends PerformanceEvaluation.TestBase>	determineCommandClass(String cmd)
(package private) static PerformanceEvaluation.RunResult[]	doLocalClients(PerformanceEvaluation.TestOptions opts, org.apache.hadoop.conf.Configuration conf)
(package private) static org.apache.hadoop.mapreduce.Job	doMapReduce(PerformanceEvaluation.TestOptions opts, org.apache.hadoop.conf.Configuration conf)
static byte[]	format(int number)
static byte[]	generateData(Random r, int length)
(package private) static int	generateRandomRow(Random random, int totalRows)
private static int	getAverageValueLength(PerformanceEvaluation.TestOptions opts)
(package private) static byte[]	getRandomRow(Random random, int totalRows)
(package private) static int	getRowsPerGB(PerformanceEvaluation.TestOptions opts)
protected static byte[][]	getSplits(PerformanceEvaluation.TestOptions opts)	generates splits based on total number of rows and specified split regions
protected static org.apache.hadoop.hbase.client.TableDescriptor	getTableDescriptor(PerformanceEvaluation.TestOptions opts)	Create an HTableDescriptor from provided TestOptions.
private static boolean	isCommandClass(String cmd)
private static boolean	isCustomTestClass(String cmd)
static void	main(String[] args)
(package private) static PerformanceEvaluation.TestOptions	parseOpts(Queue<String> args)	Parse options passed in via an arguments array.
protected void	printUsage()
protected static void	printUsage(String message)
protected static void	printUsage(String shortName, String message)
protected static void	printUsageAndExit(String message, int exitCode)
int	run(String[] args)
(package private) static PerformanceEvaluation.RunResult	runOneClient(Class<? extends PerformanceEvaluation.TestBase> cmd, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.hbase.client.Connection con, org.apache.hadoop.hbase.client.AsyncConnection asyncCon, PerformanceEvaluation.TestOptions opts, PerformanceEvaluation.Status status)
private void	runTest(Class<? extends PerformanceEvaluation.TestBase> cmd, PerformanceEvaluation.TestOptions opts)
(package private) static void	setupConnectionCount(PerformanceEvaluation.TestOptions opts)
private static void	validateParsedOpts(PerformanceEvaluation.TestOptions opts)	Validates opts after all the opts are parsed, so that caller need not to maintain order of opts
(package private) static org.apache.hadoop.fs.Path	writeInputFile(org.apache.hadoop.conf.Configuration c, PerformanceEvaluation.TestOptions opts)
(package private) static org.apache.hadoop.fs.Path	writeInputFile(org.apache.hadoop.conf.Configuration c, PerformanceEvaluation.TestOptions opts, org.apache.hadoop.fs.Path basedir)

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.conf.Configurable

getConf, setConf

Field Details

RANDOM_SEEK_SCAN

static final String RANDOM_SEEK_SCAN

See Also:

• Constant Field Values

RANDOM_READ

static final String RANDOM_READ

See Also:

• Constant Field Values

PE_COMMAND_SHORTNAME

static final String PE_COMMAND_SHORTNAME

See Also:

• Constant Field Values

LOG

private static final org.slf4j.Logger LOG

GSON

private static final org.apache.hbase.thirdparty.com.google.gson.Gson GSON

TABLE_NAME

public static final String TABLE_NAME

See Also:

• Constant Field Values

FAMILY_NAME_BASE

public static final String FAMILY_NAME_BASE

See Also:

• Constant Field Values

FAMILY_ZERO

public static final byte[] FAMILY_ZERO

COLUMN_ZERO

public static final byte[] COLUMN_ZERO

DEFAULT_VALUE_LENGTH

public static final int DEFAULT_VALUE_LENGTH

See Also:

• Constant Field Values

ROW_LENGTH

public static final int ROW_LENGTH

See Also:

• Constant Field Values

ONE_GB

private static final int ONE_GB

See Also:

• Constant Field Values

DEFAULT_ROWS_PER_GB

private static final int DEFAULT_ROWS_PER_GB

See Also:

• Constant Field Values

TAG_LENGTH

private static final int TAG_LENGTH

See Also:

• Constant Field Values

FMT

private static final DecimalFormat FMT

CXT

private static final MathContext CXT

MS_PER_SEC

private static final BigDecimal MS_PER_SEC

BYTES_PER_MB

private static final BigDecimal BYTES_PER_MB

DEFAULT_OPTS

private static final PerformanceEvaluation.TestOptions DEFAULT_OPTS

COMMANDS

private static Map<String,PerformanceEvaluation.CmdDescriptor> COMMANDS

PERF_EVAL_DIR

private static final org.apache.hadoop.fs.Path PERF_EVAL_DIR

JOB_INPUT_FILENAME

static String JOB_INPUT_FILENAME

Each client has one mapper to do the work, and client do the resulting count in a map task.

Constructor Details

PerformanceEvaluation

public PerformanceEvaluation(org.apache.hadoop.conf.Configuration conf)

Constructor

Parameters:

conf - Configuration object

Method Details

addCommandDescriptor

protected static void addCommandDescriptor(Class<? extends PerformanceEvaluation.TestBase> cmdClass,
 String name,
 String description)

checkTable

static boolean checkTable(org.apache.hadoop.hbase.client.Admin admin,
 PerformanceEvaluation.TestOptions opts)
                   throws IOException

Throws:

IOException

getTableDescriptor

protected static org.apache.hadoop.hbase.client.TableDescriptor getTableDescriptor(PerformanceEvaluation.TestOptions opts)

Create an HTableDescriptor from provided TestOptions.

getSplits

protected static byte[][] getSplits(PerformanceEvaluation.TestOptions opts)

generates splits based on total number of rows and specified split regions

setupConnectionCount

static void setupConnectionCount(PerformanceEvaluation.TestOptions opts)

doLocalClients

static PerformanceEvaluation.RunResult[] doLocalClients(PerformanceEvaluation.TestOptions opts,
 org.apache.hadoop.conf.Configuration conf)
                                                 throws IOException,
InterruptedException,
ExecutionException

Throws:

IOException

InterruptedException

ExecutionException

doMapReduce

static org.apache.hadoop.mapreduce.Job doMapReduce(PerformanceEvaluation.TestOptions opts,
 org.apache.hadoop.conf.Configuration conf)
                                            throws IOException,
InterruptedException,
ClassNotFoundException

Throws:

IOException

InterruptedException

ClassNotFoundException

writeInputFile

static org.apache.hadoop.fs.Path writeInputFile(org.apache.hadoop.conf.Configuration c,
 PerformanceEvaluation.TestOptions opts)
                                         throws IOException

Throws:

IOException

writeInputFile

static org.apache.hadoop.fs.Path writeInputFile(org.apache.hadoop.conf.Configuration c,
 PerformanceEvaluation.TestOptions opts,
 org.apache.hadoop.fs.Path basedir)
                                         throws IOException

Throws:

IOException

calculateMbps

private static String calculateMbps(int rows,
 long timeMs,
 int valueSize,
 int families,
 int columns)

Compute a throughput rate in MB/s.

Parameters:

rows - Number of records consumed.

timeMs - Time taken in milliseconds.

Returns:

String value with label, ie '123.76 MB/s'

format

public static byte[] format(int number)

generateData

public static byte[] generateData(Random r,
 int length)

getRandomRow

static byte[] getRandomRow(Random random,
 int totalRows)

generateRandomRow

static int generateRandomRow(Random random,
 int totalRows)

runOneClient

static PerformanceEvaluation.RunResult runOneClient(Class<? extends PerformanceEvaluation.TestBase> cmd,
 org.apache.hadoop.conf.Configuration conf,
 org.apache.hadoop.hbase.client.Connection con,
 org.apache.hadoop.hbase.client.AsyncConnection asyncCon,
 PerformanceEvaluation.TestOptions opts,
 PerformanceEvaluation.Status status)
                                             throws IOException,
InterruptedException

Throws:

IOException

InterruptedException

getAverageValueLength

private static int getAverageValueLength(PerformanceEvaluation.TestOptions opts)

runTest

private void runTest(Class<? extends PerformanceEvaluation.TestBase> cmd,
 PerformanceEvaluation.TestOptions opts)
              throws IOException,
InterruptedException,
ClassNotFoundException,
ExecutionException

Throws:

IOException

InterruptedException

ClassNotFoundException

ExecutionException

printUsage

protected void printUsage()

printUsage

protected static void printUsage(String message)

printUsageAndExit

protected static void printUsageAndExit(String message,
 int exitCode)

printUsage

protected static void printUsage(String shortName,
 String message)

parseOpts

static PerformanceEvaluation.TestOptions parseOpts(Queue<String> args)

Parse options passed in via an arguments array. Assumes that array has been split on white-space and placed into a Queue. Any unknown arguments will remain in the queue at the conclusion of this method call. It's up to the caller to deal with these unrecognized arguments.

validateParsedOpts

private static void validateParsedOpts(PerformanceEvaluation.TestOptions opts)

Validates opts after all the opts are parsed, so that caller need not to maintain order of opts

calculateRowsAndSize

static PerformanceEvaluation.TestOptions calculateRowsAndSize(PerformanceEvaluation.TestOptions opts)

getRowsPerGB

static int getRowsPerGB(PerformanceEvaluation.TestOptions opts)

run

public int run(String[] args)
        throws Exception

Specified by:

run in interface org.apache.hadoop.util.Tool

Throws:

Exception

isCommandClass

private static boolean isCommandClass(String cmd)

isCustomTestClass

private static boolean isCustomTestClass(String cmd)

determineCommandClass

private static Class<? extends PerformanceEvaluation.TestBase> determineCommandClass(String cmd)

main

public static void main(String[] args)
                 throws Exception

Throws:

Exception