Package org.apache.hadoop.hbase
Class PerformanceEvaluation
java.lang.Object
org.apache.hadoop.conf.Configured
org.apache.hadoop.hbase.PerformanceEvaluation
- All Implemented Interfaces:
org.apache.hadoop.conf.Configurable
,org.apache.hadoop.util.Tool
@LimitedPrivate("Tools")
public class PerformanceEvaluation
extends org.apache.hadoop.conf.Configured
implements org.apache.hadoop.util.Tool
Script used evaluating HBase performance and scalability. Runs a HBase client that steps through
one of a set of hardcoded tests or 'experiments' (e.g. a random reads test, a random writes test,
etc.). Pass on the command-line which test to run and how many clients are participating in this
experiment. Run
PerformanceEvaluation --help
to obtain usage.
This class sets up and runs the evaluation programs described in Section 7, Performance Evaluation, of the Bigtable paper, pages 8-10.
By default, runs as a mapreduce job where each mapper runs a single test client. Can also run as
a non-mapreduce, multithreaded application by specifying --nomapred
. Each client does
about 1GB of data, unless specified otherwise.
-
Nested Class Summary
Modifier and TypeClassDescription(package private) static class
(package private) static class
(package private) static class
(package private) static class
(package private) static class
(package private) static class
(package private) static class
(package private) static class
(package private) static class
(package private) static class
Base class for operations that are CAS-like; that read a value and then set it based off what they read.(package private) static class
(package private) static class
(package private) static class
(package private) static class
(package private) static class
Describes a command.protected static enum
Enum for map metrics.static class
MapReduce job that runs a performance evaluation client in each map task.(package private) static class
(package private) static class
(package private) static class
(package private) static class
(package private) static class
(package private) static class
(package private) static class
(package private) static class
(package private) static class
(package private) static class
(package private) static class
(package private) static class
(package private) static class
(package private) static class
(package private) static class
protected static class
(package private) static class
(package private) static class
(package private) static class
(package private) static class
(package private) static interface
Implementations can have their status set.(package private) static class
(package private) static class
(package private) static class
(package private) static class
Wraps up options passed toPerformanceEvaluation
. -
Field Summary
Modifier and TypeFieldDescriptionprivate static final BigDecimal
static final byte[]
private static Map<String,
PerformanceEvaluation.CmdDescriptor> private static final MathContext
private static final PerformanceEvaluation.TestOptions
private static final int
static final int
static final String
static final byte[]
private static final DecimalFormat
private static final org.apache.hbase.thirdparty.com.google.gson.Gson
(package private) static String
Each client has one mapper to do the work, and client do the resulting count in a map task.private static final org.slf4j.Logger
private static final BigDecimal
private static final int
(package private) static final String
private static final org.apache.hadoop.fs.Path
(package private) static final String
(package private) static final String
static final int
static final String
private static final int
-
Constructor Summary
ConstructorDescriptionPerformanceEvaluation
(org.apache.hadoop.conf.Configuration conf) Constructor -
Method Summary
Modifier and TypeMethodDescriptionprotected static void
addCommandDescriptor
(Class<? extends PerformanceEvaluation.TestBase> cmdClass, String name, String description) private static String
calculateMbps
(int rows, long timeMs, int valueSize, int families, int columns) Compute a throughput rate in MB/s.(package private) static PerformanceEvaluation.TestOptions
(package private) static boolean
checkTable
(Admin admin, PerformanceEvaluation.TestOptions opts) private static Class<? extends PerformanceEvaluation.TestBase>
(package private) static PerformanceEvaluation.RunResult[]
doLocalClients
(PerformanceEvaluation.TestOptions opts, org.apache.hadoop.conf.Configuration conf) (package private) static org.apache.hadoop.mapreduce.Job
doMapReduce
(PerformanceEvaluation.TestOptions opts, org.apache.hadoop.conf.Configuration conf) static byte[]
format
(int number) static byte[]
generateData
(Random r, int length) (package private) static int
generateRandomRow
(Random random, int totalRows) private static int
(package private) static byte[]
getRandomRow
(Random random, int totalRows) (package private) static int
protected static byte[][]
generates splits based on total number of rows and specified split regionsprotected static TableDescriptor
Create an HTableDescriptor from provided TestOptions.private static boolean
isCommandClass
(String cmd) private static boolean
isCustomTestClass
(String cmd) static void
(package private) static PerformanceEvaluation.TestOptions
Parse options passed in via an arguments array.protected void
protected static void
printUsage
(String message) protected static void
printUsage
(String shortName, String message) protected static void
printUsageAndExit
(String message, int exitCode) int
(package private) static PerformanceEvaluation.RunResult
runOneClient
(Class<? extends PerformanceEvaluation.TestBase> cmd, org.apache.hadoop.conf.Configuration conf, Connection con, AsyncConnection asyncCon, PerformanceEvaluation.TestOptions opts, PerformanceEvaluation.Status status) private void
runTest
(Class<? extends PerformanceEvaluation.TestBase> cmd, PerformanceEvaluation.TestOptions opts) (package private) static void
private static void
Validates opts after all the opts are parsed, so that caller need not to maintain order of opts(package private) static org.apache.hadoop.fs.Path
writeInputFile
(org.apache.hadoop.conf.Configuration c, PerformanceEvaluation.TestOptions opts) (package private) static org.apache.hadoop.fs.Path
writeInputFile
(org.apache.hadoop.conf.Configuration c, PerformanceEvaluation.TestOptions opts, org.apache.hadoop.fs.Path basedir) Methods inherited from class org.apache.hadoop.conf.Configured
getConf, setConf
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
Methods inherited from interface org.apache.hadoop.conf.Configurable
getConf, setConf
-
Field Details
-
RANDOM_SEEK_SCAN
- See Also:
-
RANDOM_READ
- See Also:
-
PE_COMMAND_SHORTNAME
- See Also:
-
LOG
-
GSON
-
TABLE_NAME
- See Also:
-
FAMILY_NAME_BASE
- See Also:
-
FAMILY_ZERO
-
COLUMN_ZERO
-
DEFAULT_VALUE_LENGTH
- See Also:
-
ROW_LENGTH
- See Also:
-
ONE_GB
- See Also:
-
DEFAULT_ROWS_PER_GB
- See Also:
-
TAG_LENGTH
- See Also:
-
FMT
-
CXT
-
MS_PER_SEC
-
BYTES_PER_MB
-
DEFAULT_OPTS
-
COMMANDS
-
PERF_EVAL_DIR
-
JOB_INPUT_FILENAME
Each client has one mapper to do the work, and client do the resulting count in a map task.
-
-
Constructor Details
-
PerformanceEvaluation
Constructor- Parameters:
conf
- Configuration object
-
-
Method Details
-
addCommandDescriptor
protected static void addCommandDescriptor(Class<? extends PerformanceEvaluation.TestBase> cmdClass, String name, String description) -
checkTable
- Throws:
IOException
-
getTableDescriptor
Create an HTableDescriptor from provided TestOptions. -
getSplits
generates splits based on total number of rows and specified split regions -
setupConnectionCount
-
doLocalClients
static PerformanceEvaluation.RunResult[] doLocalClients(PerformanceEvaluation.TestOptions opts, org.apache.hadoop.conf.Configuration conf) throws IOException, InterruptedException, ExecutionException -
doMapReduce
static org.apache.hadoop.mapreduce.Job doMapReduce(PerformanceEvaluation.TestOptions opts, org.apache.hadoop.conf.Configuration conf) throws IOException, InterruptedException, ClassNotFoundException -
writeInputFile
static org.apache.hadoop.fs.Path writeInputFile(org.apache.hadoop.conf.Configuration c, PerformanceEvaluation.TestOptions opts) throws IOException - Throws:
IOException
-
writeInputFile
static org.apache.hadoop.fs.Path writeInputFile(org.apache.hadoop.conf.Configuration c, PerformanceEvaluation.TestOptions opts, org.apache.hadoop.fs.Path basedir) throws IOException - Throws:
IOException
-
calculateMbps
private static String calculateMbps(int rows, long timeMs, int valueSize, int families, int columns) Compute a throughput rate in MB/s.- Parameters:
rows
- Number of records consumed.timeMs
- Time taken in milliseconds.- Returns:
- String value with label, ie '123.76 MB/s'
-
format
-
generateData
-
getRandomRow
-
generateRandomRow
-
runOneClient
static PerformanceEvaluation.RunResult runOneClient(Class<? extends PerformanceEvaluation.TestBase> cmd, org.apache.hadoop.conf.Configuration conf, Connection con, AsyncConnection asyncCon, PerformanceEvaluation.TestOptions opts, PerformanceEvaluation.Status status) throws IOException, InterruptedException - Throws:
IOException
InterruptedException
-
getAverageValueLength
-
runTest
private void runTest(Class<? extends PerformanceEvaluation.TestBase> cmd, PerformanceEvaluation.TestOptions opts) throws IOException, InterruptedException, ClassNotFoundException, ExecutionException -
printUsage
-
printUsage
-
printUsageAndExit
-
printUsage
-
parseOpts
Parse options passed in via an arguments array. Assumes that array has been split on white-space and placed into aQueue
. Any unknown arguments will remain in the queue at the conclusion of this method call. It's up to the caller to deal with these unrecognized arguments. -
validateParsedOpts
Validates opts after all the opts are parsed, so that caller need not to maintain order of opts -
calculateRowsAndSize
-
getRowsPerGB
-
run
- Specified by:
run
in interfaceorg.apache.hadoop.util.Tool
- Throws:
Exception
-
isCommandClass
-
isCustomTestClass
-
determineCommandClass
-
main
- Throws:
Exception
-