org.apache.hadoop.hbase.regionserver

Class DataBlockEncodingTool

java.lang.Object

org.apache.hadoop.hbase.regionserver.DataBlockEncodingTool

public class DataBlockEncodingTool
extends Object

Tests various algorithms for key compression on an existing HFile. Useful for testing, debugging and benchmarking.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
private static class	DataBlockEncodingTool.Manipulation

Field Summary

Fields

Modifier and Type	Field and Description
private static int	benchmarkNOmit
private static int	benchmarkNTimes
private static double	BYTES_IN_MB
private List<org.apache.hadoop.hbase.io.encoding.EncodedDataBlock>	codecs
private org.apache.hadoop.hbase.io.compress.Compression.Algorithm	compressionAlgorithm
private String	compressionAlgorithmName
private org.apache.hadoop.io.compress.Compressor	compressor
private org.apache.hadoop.conf.Configuration	conf
private org.apache.hadoop.io.compress.Decompressor	decompressor
private static int	DEFAULT_BENCHMARK_N_OMIT How many first runs should not be included in the benchmark.
private static int	DEFAULT_BENCHMARK_N_TIMES How many times to run the benchmark.
private static org.apache.hadoop.hbase.io.compress.Compression.Algorithm	DEFAULT_COMPRESSION Compression algorithm to use if not specified on the command line
private static DecimalFormat	DELIMITED_DECIMAL_FORMAT
private static boolean	includesMemstoreTS
private static String	INT_FORMAT
private static org.slf4j.Logger	LOG
private static double	MB_SEC_COEF
private static double	NS_IN_SEC
private static String	OPT_BENCHMARK_N_OMIT Number of first runs of every benchmark to omit from statistics
private static String	OPT_BENCHMARK_N_TIMES Number of times to run each benchmark
private static String	OPT_COMPRESSION_ALGORITHM What compression algorithm to test
private static String	OPT_HFILE_NAME HFile name to be used in benchmark
private static String	OPT_KV_LIMIT Maximum number of key/value pairs to process in a single benchmark run
private static String	OPT_MEASURE_THROUGHPUT Whether to run a benchmark to measure read throughput
private static String	OPT_OMIT_CORRECTNESS_TEST If this is specified, no correctness testing will be done
private static String	PCT_FORMAT
private byte[]	rawKVs
private long	totalCFLength
private long	totalKeyLength
private long	totalKeyRedundancyLength
private long	totalPrefixLength
private long	totalValueLength
private static boolean	USE_TAG
private boolean	useHBaseChecksum

Constructor Summary

Constructors

Constructor and Description
DataBlockEncodingTool(org.apache.hadoop.conf.Configuration conf, String compressionAlgorithmName)

Method Summary

Modifier and Type	Method and Description
void	benchmarkAlgorithm(org.apache.hadoop.hbase.io.compress.Compression.Algorithm algorithm, String name, byte[] buffer, int offset, int length) Check decompress performance of a given algorithm and print it.
void	benchmarkCodecs() Benchmark codec's speed.
private void	benchmarkDefaultCompression(int totalSize, byte[] rawBuffer)
private int	benchmarkEncoder(int previousTotalSize, org.apache.hadoop.hbase.io.encoding.EncodedDataBlock codec) Benchmark compression/decompression throughput.
void	checkStatistics(org.apache.hadoop.hbase.regionserver.KeyValueScanner scanner, int kvLimit) Check statistics for given HFile for different data block encoders.
void	displayStatistics() Display statistics of different compression algorithms.
static void	main(String[] args) A command line interface to benchmarks.
private void	outputSavings(String caption, long part, long whole)
private static void	outputTuple(String caption, String format, Object... values)
private void	outputTuplePct(String caption, long size)
private static void	printBenchmarkResult(int totalSize, List<Long> durationsInNanoSec, DataBlockEncodingTool.Manipulation manipulation)
private static void	printUsage(org.apache.hbase.thirdparty.org.apache.commons.cli.Options options)
static void	testCodecs(org.apache.hadoop.conf.Configuration conf, int kvLimit, String hfilePath, String compressionName, boolean doBenchmark, boolean doVerify) Test a data block encoder on the given HFile.
boolean	verifyCodecs(org.apache.hadoop.hbase.regionserver.KeyValueScanner scanner, int kvLimit) Verify if all data block encoders are working properly.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

private static final org.slf4j.Logger LOG

includesMemstoreTS

private static final boolean includesMemstoreTS

See Also:

Constant Field Values

DEFAULT_BENCHMARK_N_TIMES

private static final int DEFAULT_BENCHMARK_N_TIMES

How many times to run the benchmark. More times means better data in terms of statistics but slower execution. Has to be strictly larger than DEFAULT_BENCHMARK_N_OMIT.

See Also:

Constant Field Values

DEFAULT_BENCHMARK_N_OMIT

private static final int DEFAULT_BENCHMARK_N_OMIT

How many first runs should not be included in the benchmark. Done in order to exclude setup cost.

See Also:

Constant Field Values

OPT_HFILE_NAME

private static final String OPT_HFILE_NAME

HFile name to be used in benchmark

See Also:

Constant Field Values

OPT_KV_LIMIT

private static final String OPT_KV_LIMIT

Maximum number of key/value pairs to process in a single benchmark run

See Also:

Constant Field Values

OPT_MEASURE_THROUGHPUT

private static final String OPT_MEASURE_THROUGHPUT

Whether to run a benchmark to measure read throughput

See Also:

Constant Field Values

OPT_OMIT_CORRECTNESS_TEST

private static final String OPT_OMIT_CORRECTNESS_TEST

If this is specified, no correctness testing will be done

See Also:

Constant Field Values

OPT_COMPRESSION_ALGORITHM

private static final String OPT_COMPRESSION_ALGORITHM

What compression algorithm to test

See Also:

Constant Field Values

OPT_BENCHMARK_N_TIMES

private static final String OPT_BENCHMARK_N_TIMES

Number of times to run each benchmark

See Also:

Constant Field Values

OPT_BENCHMARK_N_OMIT

private static final String OPT_BENCHMARK_N_OMIT

Number of first runs of every benchmark to omit from statistics

See Also:

Constant Field Values

DEFAULT_COMPRESSION

private static final org.apache.hadoop.hbase.io.compress.Compression.Algorithm DEFAULT_COMPRESSION

Compression algorithm to use if not specified on the command line

DELIMITED_DECIMAL_FORMAT

private static final DecimalFormat DELIMITED_DECIMAL_FORMAT

PCT_FORMAT

private static final String PCT_FORMAT

See Also:

Constant Field Values

INT_FORMAT

private static final String INT_FORMAT

See Also:

Constant Field Values

benchmarkNTimes

private static int benchmarkNTimes

benchmarkNOmit

private static int benchmarkNOmit

conf

private final org.apache.hadoop.conf.Configuration conf

codecs

private List<org.apache.hadoop.hbase.io.encoding.EncodedDataBlock> codecs

totalPrefixLength

private long totalPrefixLength

totalKeyLength

private long totalKeyLength

totalValueLength

private long totalValueLength

totalKeyRedundancyLength

private long totalKeyRedundancyLength

totalCFLength

private long totalCFLength

rawKVs

private byte[] rawKVs

useHBaseChecksum

private boolean useHBaseChecksum

compressionAlgorithmName

private final String compressionAlgorithmName

compressionAlgorithm

private final org.apache.hadoop.hbase.io.compress.Compression.Algorithm compressionAlgorithm

compressor

private final org.apache.hadoop.io.compress.Compressor compressor

decompressor

private final org.apache.hadoop.io.compress.Decompressor decompressor

USE_TAG

private static boolean USE_TAG

BYTES_IN_MB

private static final double BYTES_IN_MB

See Also:

Constant Field Values

NS_IN_SEC

private static final double NS_IN_SEC

See Also:

Constant Field Values

MB_SEC_COEF

private static final double MB_SEC_COEF

See Also:

Constant Field Values

Constructor Detail

DataBlockEncodingTool

public DataBlockEncodingTool(org.apache.hadoop.conf.Configuration conf,
                             String compressionAlgorithmName)

Parameters:

compressionAlgorithmName - What kind of algorithm should be used as baseline for comparison (e.g. lzo, gz).

Method Detail

checkStatistics

public void checkStatistics(org.apache.hadoop.hbase.regionserver.KeyValueScanner scanner,
                            int kvLimit)
                     throws IOException

Check statistics for given HFile for different data block encoders.

Parameters:

scanner - Of file which will be compressed.

kvLimit - Maximal count of KeyValue which will be processed.

Throws:

IOException - thrown if scanner is invalid

verifyCodecs

public boolean verifyCodecs(org.apache.hadoop.hbase.regionserver.KeyValueScanner scanner,
                            int kvLimit)
                     throws IOException

Verify if all data block encoders are working properly.

Parameters:

scanner - Of file which was compressed.

kvLimit - Maximal count of KeyValue which will be processed.

Returns:

true if all data block encoders compressed/decompressed correctly.

Throws:

IOException - thrown if scanner is invalid

benchmarkCodecs

public void benchmarkCodecs()
                     throws IOException

Benchmark codec's speed.

Throws:

IOException

benchmarkEncoder

private int benchmarkEncoder(int previousTotalSize,
                             org.apache.hadoop.hbase.io.encoding.EncodedDataBlock codec)

Benchmark compression/decompression throughput.

Parameters:

previousTotalSize - Total size used for verification. Use -1 if unknown.

codec - Tested encoder.

Returns:

Size of uncompressed data.

benchmarkDefaultCompression

private void benchmarkDefaultCompression(int totalSize,
                                         byte[] rawBuffer)
                                  throws IOException

Throws:

IOException

benchmarkAlgorithm

public void benchmarkAlgorithm(org.apache.hadoop.hbase.io.compress.Compression.Algorithm algorithm,
                               String name,
                               byte[] buffer,
                               int offset,
                               int length)
                        throws IOException

Check decompress performance of a given algorithm and print it.

Parameters:

algorithm - Compression algorithm.

name - Name of algorithm.

buffer - Buffer to be compressed.

offset - Position of the beginning of the data.

length - Length of data in buffer. n

Throws:

IOException

printBenchmarkResult

private static void printBenchmarkResult(int totalSize,
                                         List<Long> durationsInNanoSec,
                                         DataBlockEncodingTool.Manipulation manipulation)

outputTuple

private static void outputTuple(String caption,
                                String format,
                                Object... values)

displayStatistics

public void displayStatistics()
                       throws IOException

Display statistics of different compression algorithms. n

Throws:

IOException

outputTuplePct

private void outputTuplePct(String caption,
                            long size)

outputSavings

private void outputSavings(String caption,
                           long part,
                           long whole)

testCodecs

public static void testCodecs(org.apache.hadoop.conf.Configuration conf,
                              int kvLimit,
                              String hfilePath,
                              String compressionName,
                              boolean doBenchmark,
                              boolean doVerify)
                       throws IOException

Test a data block encoder on the given HFile. Output results to console.

Parameters:

kvLimit - The limit of KeyValue which will be analyzed.

hfilePath - an HFile path on the file system.

compressionName - Compression algorithm used for comparison.

doBenchmark - Run performance benchmarks.

doVerify - Verify correctness.

Throws:

IOException - When pathName is incorrect.

printUsage

private static void printUsage(org.apache.hbase.thirdparty.org.apache.commons.cli.Options options)

main

public static void main(String[] args)
                 throws IOException

A command line interface to benchmarks. Parses command-line arguments and runs the appropriate benchmarks.

Parameters:

args - Should have length at least 1 and holds the file path to HFile.

Throws:

IOException - If you specified the wrong file.