org.apache.hadoop.hbase.rest

Class PerformanceEvaluation

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.hbase.rest.PerformanceEvaluation

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, org.apache.hadoop.util.Tool

public class PerformanceEvaluation
extends org.apache.hadoop.conf.Configured
implements org.apache.hadoop.util.Tool

Script used evaluating Stargate performance and scalability. Runs a SG client that steps through one of a set of hardcoded tests or 'experiments' (e.g. a random reads test, a random writes test, etc.). Pass on the command-line which test to run and how many clients are participating in this experiment. Run java PerformanceEvaluation --help to obtain usage.

This class sets up and runs the evaluation programs described in Section 7, Performance Evaluation, of the Bigtable paper, pages 8-10.

If number of clients > 1, we start up a MapReduce job. Each map task runs an individual client. Each client does about 1GB of data.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
(package private) static class	PerformanceEvaluation.BufferedMutatorTest
(package private) static class	PerformanceEvaluation.CmdDescriptor Describes a command.
protected static class	PerformanceEvaluation.Counter Enum for map metrics.
static class	PerformanceEvaluation.EvaluationMapTask MapReduce job that runs a performance evaluation client in each map task.
(package private) static class	PerformanceEvaluation.FilteredScanTest
static class	PerformanceEvaluation.PeInputFormat InputFormat of Performance Evaluation MapReduce job.
static class	PerformanceEvaluation.PeInputSplit This class works as the InputSplit of Performance Evaluation MapReduce InputFormat, and the Record Value of RecordReader.
(package private) static class	PerformanceEvaluation.RandomReadTest
(package private) static class	PerformanceEvaluation.RandomScanWithRange10000Test
(package private) static class	PerformanceEvaluation.RandomScanWithRange1000Test
(package private) static class	PerformanceEvaluation.RandomScanWithRange100Test
(package private) static class	PerformanceEvaluation.RandomScanWithRange10Test
(package private) static class	PerformanceEvaluation.RandomScanWithRangeTest
(package private) static class	PerformanceEvaluation.RandomSeekScanTest
(package private) static class	PerformanceEvaluation.RandomWriteTest
(package private) static class	PerformanceEvaluation.ScanTest
(package private) static class	PerformanceEvaluation.SequentialReadTest
(package private) static class	PerformanceEvaluation.SequentialWriteTest
(package private) static interface	PerformanceEvaluation.Status Implementations can have their status set.
(package private) static class	PerformanceEvaluation.TableTest
(package private) static class	PerformanceEvaluation.Test
(package private) static class	PerformanceEvaluation.TestOptions Wraps up options passed to PerformanceEvaluation tests This makes the reflection logic a little easier to understand...

Field Summary

Fields

Modifier and Type	Field and Description
private org.apache.hadoop.hbase.io.encoding.DataBlockEncoding	blockEncoding
protected static org.apache.hadoop.hbase.rest.client.Cluster	cluster
protected Map<String,PerformanceEvaluation.CmdDescriptor>	commands
private org.apache.hadoop.hbase.io.compress.Compression.Algorithm	compression
(package private) org.apache.hadoop.conf.Configuration	conf
private org.apache.hadoop.hbase.client.Connection	connection
private static int	DEFAULT_ROW_PREFIX_LENGTH
static byte[]	FAMILY_NAME
private boolean	flushCommits
private boolean	inMemoryCF
static Pattern	LINE_PATTERN Regex to parse lines in input file passed to mapreduce task.
protected static org.slf4j.Logger	LOG
private int	N
private boolean	nomapred
private int	noOfTags
private static int	ONE_GB
private static org.apache.hadoop.fs.Path	PERF_EVAL_DIR
private int	presplitRegions
static byte[]	QUALIFIER_NAME
private int	R
private static int	ROW_LENGTH
private static int	ROWS_PER_GB
protected org.apache.hadoop.hbase.HTableDescriptor	TABLE_DESCRIPTOR
static org.apache.hadoop.hbase.TableName	TABLE_NAME
private org.apache.hadoop.hbase.TableName	tableName
private static int	TAG_LENGTH
private boolean	useTags
private boolean	writeToWAL

Constructor Summary

Constructors

Constructor and Description
PerformanceEvaluation(org.apache.hadoop.conf.Configuration c) Constructor

Method Summary

Modifier and Type	Method and Description
protected void	addCommandDescriptor(Class<? extends PerformanceEvaluation.Test> cmdClass, String name, String description)
private boolean	checkTable(RemoteAdmin admin) If table does not already exist, create.
private Class<? extends PerformanceEvaluation.Test>	determineCommandClass(String cmd)
private void	doMapReduce(Class<? extends PerformanceEvaluation.Test> cmd) Run a mapreduce job.
private void	doMultipleClients(Class<? extends PerformanceEvaluation.Test> cmd) Run all clients in this vm each to its own thread.
static byte[]	format(int number) Format passed integer.
static byte[]	generateData(Random r, int length)
static byte[]	generateValue(Random r)
private void	getArgs(int start, String[] args)
(package private) static byte[]	getRandomRow(Random random, int totalRows)
protected byte[][]	getSplits() Generates splits based on total number of rows and specified split regions
protected org.apache.hadoop.hbase.HTableDescriptor	getTableDescriptor()
static void	main(String[] args)
protected void	printUsage()
protected void	printUsage(String message)
int	run(String[] args)
private void	runNIsMoreThanOne(Class<? extends PerformanceEvaluation.Test> cmd) We're to run multiple clients concurrently.
private void	runNIsOne(Class<? extends PerformanceEvaluation.Test> cmd)
(package private) long	runOneClient(Class<? extends PerformanceEvaluation.Test> cmd, int startRow, int perClientRunRows, int totalRows, boolean flushCommits, boolean writeToWAL, boolean useTags, int noOfTags, org.apache.hadoop.hbase.client.Connection connection, PerformanceEvaluation.Status status)
private void	runTest(Class<? extends PerformanceEvaluation.Test> cmd)
private org.apache.hadoop.fs.Path	writeInputFile(org.apache.hadoop.conf.Configuration c) Write input file of offsets-per-client for the mapreduce job.

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.conf.Configurable

getConf, setConf

Field Detail

LOG

protected static final org.slf4j.Logger LOG

DEFAULT_ROW_PREFIX_LENGTH

private static final int DEFAULT_ROW_PREFIX_LENGTH

See Also:

Constant Field Values

ROW_LENGTH

private static final int ROW_LENGTH

See Also:

Constant Field Values

TAG_LENGTH

private static final int TAG_LENGTH

See Also:

Constant Field Values

ONE_GB

private static final int ONE_GB

See Also:

Constant Field Values

ROWS_PER_GB

private static final int ROWS_PER_GB

See Also:

Constant Field Values

TABLE_NAME

public static final org.apache.hadoop.hbase.TableName TABLE_NAME

FAMILY_NAME

public static final byte[] FAMILY_NAME

QUALIFIER_NAME

public static final byte[] QUALIFIER_NAME

tableName

private org.apache.hadoop.hbase.TableName tableName

TABLE_DESCRIPTOR

protected org.apache.hadoop.hbase.HTableDescriptor TABLE_DESCRIPTOR

commands

protected Map<String,PerformanceEvaluation.CmdDescriptor> commands

cluster

protected static org.apache.hadoop.hbase.rest.client.Cluster cluster

conf

volatile org.apache.hadoop.conf.Configuration conf

nomapred

private boolean nomapred

N

private int N

R

private int R

compression

private org.apache.hadoop.hbase.io.compress.Compression.Algorithm compression

blockEncoding

private org.apache.hadoop.hbase.io.encoding.DataBlockEncoding blockEncoding

flushCommits

private boolean flushCommits

writeToWAL

private boolean writeToWAL

inMemoryCF

private boolean inMemoryCF

presplitRegions

private int presplitRegions

useTags

private boolean useTags

noOfTags

private int noOfTags

connection

private org.apache.hadoop.hbase.client.Connection connection

PERF_EVAL_DIR

private static final org.apache.hadoop.fs.Path PERF_EVAL_DIR

LINE_PATTERN

public static final Pattern LINE_PATTERN

Regex to parse lines in input file passed to mapreduce task.

Constructor Detail

PerformanceEvaluation

public PerformanceEvaluation(org.apache.hadoop.conf.Configuration c)

Constructor

Parameters:

c - Configuration object

Method Detail

addCommandDescriptor

protected void addCommandDescriptor(Class<? extends PerformanceEvaluation.Test> cmdClass,
                                    String name,
                                    String description)

checkTable

private boolean checkTable(RemoteAdmin admin)
                    throws IOException

If table does not already exist, create.

Parameters:

admin - Client to use checking.

Returns:

True if we created the table.

Throws:

IOException - if an operation on the table fails

getTableDescriptor

protected org.apache.hadoop.hbase.HTableDescriptor getTableDescriptor()

getSplits

protected byte[][] getSplits()

Generates splits based on total number of rows and specified split regions

Returns:

splits : array of byte []

runNIsMoreThanOne

private void runNIsMoreThanOne(Class<? extends PerformanceEvaluation.Test> cmd)
                        throws IOException,
                               InterruptedException,
                               ClassNotFoundException

We're to run multiple clients concurrently. Setup a mapreduce job. Run one map per client. Then run a single reduce to sum the elapsed times.

Parameters:

cmd - Command to run.

Throws:

IOException

InterruptedException

ClassNotFoundException

doMultipleClients

private void doMultipleClients(Class<? extends PerformanceEvaluation.Test> cmd)
                        throws IOException

Run all clients in this vm each to its own thread.

Parameters:

cmd - Command to run

Throws:

IOException - if creating a connection fails

doMapReduce

private void doMapReduce(Class<? extends PerformanceEvaluation.Test> cmd)
                  throws IOException,
                         InterruptedException,
                         ClassNotFoundException

Run a mapreduce job. Run as many maps as asked-for clients. Before we start up the job, write out an input file with instruction per client regards which row they are to start on.

Parameters:

cmd - Command to run.

Throws:

IOException

InterruptedException

ClassNotFoundException

writeInputFile

private org.apache.hadoop.fs.Path writeInputFile(org.apache.hadoop.conf.Configuration c)
                                          throws IOException

Write input file of offsets-per-client for the mapreduce job.

Parameters:

c - Configuration

Returns:

Directory that contains file written.

Throws:

IOException - if creating the directory or the file fails

format

public static byte[] format(int number)

Format passed integer.

Parameters:

number - the integer to format

Returns:

Returns zero-prefixed 10-byte wide decimal version of passed number (Does absolute in case number is negative).

generateData

public static byte[] generateData(Random r,
                                  int length)

generateValue

public static byte[] generateValue(Random r)

getRandomRow

static byte[] getRandomRow(Random random,
                           int totalRows)

runOneClient

long runOneClient(Class<? extends PerformanceEvaluation.Test> cmd,
                  int startRow,
                  int perClientRunRows,
                  int totalRows,
                  boolean flushCommits,
                  boolean writeToWAL,
                  boolean useTags,
                  int noOfTags,
                  org.apache.hadoop.hbase.client.Connection connection,
                  PerformanceEvaluation.Status status)
           throws IOException

Throws:

IOException

runNIsOne

private void runNIsOne(Class<? extends PerformanceEvaluation.Test> cmd)

runTest

private void runTest(Class<? extends PerformanceEvaluation.Test> cmd)
              throws IOException,
                     InterruptedException,
                     ClassNotFoundException

Throws:

IOException

InterruptedException

ClassNotFoundException

printUsage

protected void printUsage()

printUsage

protected void printUsage(String message)

getArgs

private void getArgs(int start,
                     String[] args)

run

public int run(String[] args)
        throws Exception

Specified by:

run in interface org.apache.hadoop.util.Tool

Throws:

Exception

determineCommandClass

private Class<? extends PerformanceEvaluation.Test> determineCommandClass(String cmd)

main

public static void main(String[] args)
                 throws Exception

Throws:

Exception