org.apache.hadoop.hbase.util

Class RedundantKVGenerator

java.lang.Object

org.apache.hadoop.hbase.util.RedundantKVGenerator

@InterfaceAudience.Private
public class RedundantKVGenerator
extends Object

Generate list of key values which are very useful to test data block encoding and compression.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
(package private) static class	RedundantKVGenerator.ExtendedOffheapKeyValue

Field Summary

Fields

Modifier and Type	Field and Description
private int	averagePrefixLength
private int	averageQualifierLength
private int	averageSuffixLength
private int	baseTimestampDivide
private float	chanceForSameQualifier
private float	chanceForSimilarQualifier
private float	chanceForZeroValue
private int	columnFamilyLength
private byte[]	commonPrefix
(package private) static int	DEFAULT_AVERAGE_PREFIX_LENGTH
(package private) static int	DEFAULT_AVERAGE_QUALIFIER_LENGTH
(package private) static int	DEFAULT_AVERAGE_SUFFIX_LENGTH
(package private) static int	DEFAULT_BASE_TIMESTAMP_DIVIDE
(package private) static float	DEFAULT_CHANCE_FOR_SAME_QUALIFIER
(package private) static float	DEFAULT_CHANCE_FOR_SIMILIAR_QUALIFIER
(package private) static float	DEFAULT_CHANCE_FOR_ZERO_VALUE
(package private) static int	DEFAULT_COLUMN_FAMILY_LENGTH
(package private) static byte[]	DEFAULT_COMMON_PREFIX
(package private) static int	DEFAULT_NUMBER_OF_ROW
(package private) static int	DEFAULT_NUMBER_OF_ROW_PREFIXES
(package private) static int	DEFAULT_PREFIX_LENGTH_VARIANCE
(package private) static int	DEFAULT_QUALIFIER_LENGTH_VARIANCE
(package private) static int	DEFAULT_SUFFIX_LENGTH_VARIANCE
(package private) static int	DEFAULT_TIMESTAMP_DIFF_SIZE
(package private) static int	DEFAULT_VALUE_LENGTH
private byte[]	family
private int	numberOfRowPrefixes
private int	numberOfRows
private int	prefixLengthVariance
private int	qualifierLengthVariance
private Random	randomizer Used to generate dataset
private int	suffixLengthVariance
private int	timestampDiffSize
private int	valueLength

Constructor Summary

Constructors

Constructor and Description
RedundantKVGenerator() Default constructor, assumes all parameters from class constants.
RedundantKVGenerator(Random randomizer, int numberOfRowPrefixes, int averagePrefixLength, int prefixLengthVariance, int averageSuffixLength, int suffixLengthVariance, int numberOfRows, float chanceForSameQualifier, float chanceForSimiliarQualifier, int averageQualifierLength, int qualifierLengthVariance, int columnFamilyLength, int valueLength, float chanceForZeroValue, int baseTimestampDivide, int timestampDiffSize) Various configuration options for generating key values

Method Summary

Modifier and Type	Method and Description
static ByteBuffer	convertKvToByteBuffer(List<org.apache.hadoop.hbase.KeyValue> keyValues, boolean includesMemstoreTS) Convert list of KeyValues to byte buffer.
private List<byte[]>	generateRows()
List<org.apache.hadoop.hbase.Cell>	generateTestExtendedOffheapKeyValues(int howMany, boolean useTags) Generate test data useful to test encoders.
List<org.apache.hadoop.hbase.KeyValue>	generateTestKeyValues(int howMany) Generate test data useful to test encoders.
List<org.apache.hadoop.hbase.KeyValue>	generateTestKeyValues(int howMany, boolean useTags) Generate test data useful to test encoders.
RedundantKVGenerator	setFamily(byte[] family)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_COMMON_PREFIX

static byte[] DEFAULT_COMMON_PREFIX

DEFAULT_NUMBER_OF_ROW_PREFIXES

static int DEFAULT_NUMBER_OF_ROW_PREFIXES

DEFAULT_AVERAGE_PREFIX_LENGTH

static int DEFAULT_AVERAGE_PREFIX_LENGTH

DEFAULT_PREFIX_LENGTH_VARIANCE

static int DEFAULT_PREFIX_LENGTH_VARIANCE

DEFAULT_AVERAGE_SUFFIX_LENGTH

static int DEFAULT_AVERAGE_SUFFIX_LENGTH

DEFAULT_SUFFIX_LENGTH_VARIANCE

static int DEFAULT_SUFFIX_LENGTH_VARIANCE

DEFAULT_NUMBER_OF_ROW

static int DEFAULT_NUMBER_OF_ROW

DEFAULT_CHANCE_FOR_SAME_QUALIFIER

static float DEFAULT_CHANCE_FOR_SAME_QUALIFIER

DEFAULT_CHANCE_FOR_SIMILIAR_QUALIFIER

static float DEFAULT_CHANCE_FOR_SIMILIAR_QUALIFIER

DEFAULT_AVERAGE_QUALIFIER_LENGTH

static int DEFAULT_AVERAGE_QUALIFIER_LENGTH

DEFAULT_QUALIFIER_LENGTH_VARIANCE

static int DEFAULT_QUALIFIER_LENGTH_VARIANCE

DEFAULT_COLUMN_FAMILY_LENGTH

static int DEFAULT_COLUMN_FAMILY_LENGTH

DEFAULT_VALUE_LENGTH

static int DEFAULT_VALUE_LENGTH

DEFAULT_CHANCE_FOR_ZERO_VALUE

static float DEFAULT_CHANCE_FOR_ZERO_VALUE

DEFAULT_BASE_TIMESTAMP_DIVIDE

static int DEFAULT_BASE_TIMESTAMP_DIVIDE

DEFAULT_TIMESTAMP_DIFF_SIZE

static int DEFAULT_TIMESTAMP_DIFF_SIZE

randomizer

private Random randomizer

Used to generate dataset

commonPrefix

private byte[] commonPrefix

numberOfRowPrefixes

private int numberOfRowPrefixes

averagePrefixLength

private int averagePrefixLength

prefixLengthVariance

private int prefixLengthVariance

averageSuffixLength

private int averageSuffixLength

suffixLengthVariance

private int suffixLengthVariance

numberOfRows

private int numberOfRows

family

private byte[] family

chanceForSameQualifier

private float chanceForSameQualifier

chanceForSimilarQualifier

private float chanceForSimilarQualifier

averageQualifierLength

private int averageQualifierLength

qualifierLengthVariance

private int qualifierLengthVariance

columnFamilyLength

private int columnFamilyLength

valueLength

private int valueLength

chanceForZeroValue

private float chanceForZeroValue

baseTimestampDivide

private int baseTimestampDivide

timestampDiffSize

private int timestampDiffSize

Constructor Detail

RedundantKVGenerator

public RedundantKVGenerator()

Default constructor, assumes all parameters from class constants.

RedundantKVGenerator

public RedundantKVGenerator(Random randomizer,
                            int numberOfRowPrefixes,
                            int averagePrefixLength,
                            int prefixLengthVariance,
                            int averageSuffixLength,
                            int suffixLengthVariance,
                            int numberOfRows,
                            float chanceForSameQualifier,
                            float chanceForSimiliarQualifier,
                            int averageQualifierLength,
                            int qualifierLengthVariance,
                            int columnFamilyLength,
                            int valueLength,
                            float chanceForZeroValue,
                            int baseTimestampDivide,
                            int timestampDiffSize)

Various configuration options for generating key values

Parameters:

randomizer - pick things by random

Method Detail

generateRows

private List<byte[]> generateRows()

generateTestKeyValues

public List<org.apache.hadoop.hbase.KeyValue> generateTestKeyValues(int howMany)

Generate test data useful to test encoders.

Parameters:

howMany - How many Key values should be generated.

Returns:

sorted list of key values

generateTestKeyValues

public List<org.apache.hadoop.hbase.KeyValue> generateTestKeyValues(int howMany,
                                                                    boolean useTags)

Generate test data useful to test encoders.

Parameters:

howMany - How many Key values should be generated.

Returns:

sorted list of key values

generateTestExtendedOffheapKeyValues

public List<org.apache.hadoop.hbase.Cell> generateTestExtendedOffheapKeyValues(int howMany,
                                                                               boolean useTags)

Generate test data useful to test encoders.

Parameters:

howMany - How many Key values should be generated.

Returns:

sorted list of key values

convertKvToByteBuffer

public static ByteBuffer convertKvToByteBuffer(List<org.apache.hadoop.hbase.KeyValue> keyValues,
                                               boolean includesMemstoreTS)

Convert list of KeyValues to byte buffer.

Parameters:

keyValues - list of KeyValues to be converted.

Returns:

buffer with content from key values

setFamily

public RedundantKVGenerator setFamily(byte[] family)