org.apache.hadoop.hbase.regionserver

Class TestMultiColumnScanner

java.lang.Object

org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner

Direct Known Subclasses:

TestMultiColumnScannerWithAlgoGZAndNoDataEncoding, TestMultiColumnScannerWithAlgoGZAndUseDataEncoding, TestMultiColumnScannerWithNoneAndNoDataEncoding, TestMultiColumnScannerWithNoneAndUseDataEncoding

public abstract class TestMultiColumnScanner
extends Object

Tests optimized scanning of multiple columns.
We separated the big test into several sub-class UT, because When in ROWCOL bloom type, we will test the row-col bloom filter frequently for saving HDFS seek once we switch from one column to another in our UT. It's cpu time consuming (~45s for each case), so moved the ROWCOL case into a separated LargeTests to avoid timeout failure.

To be clear: In TestMultiColumnScanner, we will flush 10 (NUM_FLUSHES=10) HFiles here, and the table will put ~1000 cells (rows=20, ts=6, qualifiers=8, total=20*6*8 ~ 1000) . Each full table scan will check the ROWCOL bloom filter 20 (rows)* 8 (column) * 10 (hfiles)= 1600 times, beside it will scan the full table 6*2^8=1536 times, so finally will have 1600*1536=2457600 bloom filter testing. (See HBASE-21520)

Field Summary

Fields

Modifier and Type	Field and Description
private static long	BIG_LONG A large value of type long for use as a timestamp
org.apache.hadoop.hbase.regionserver.BloomType	bloomType
private static double	COLUMN_SKIP_IN_STORE_FILE_PROB The probability that a column is skipped in a store file.
org.apache.hadoop.hbase.io.compress.Compression.Algorithm	comprAlgo
org.apache.hadoop.hbase.io.encoding.DataBlockEncoding	dataBlockEncoding
private static double	DELETE_PROBABILITY The probability to delete a row/column pair
private static String	FAMILY
private static byte[]	FAMILY_BYTES
private static org.slf4j.Logger	LOG
private static int	MAX_COLUMN_BIT_MASK
(package private) static int	MAX_VERSIONS
private static int	NUM_COLUMNS The size of the column qualifier set used.
private static int	NUM_FLUSHES
private static int	NUM_ROWS
private static String	TABLE_NAME
private static HBaseTestingUtility	TEST_UTIL
private static long[]	TIMESTAMPS Timestamps to test with.

Constructor Summary

Constructors

Constructor and Description
TestMultiColumnScanner()

Method Summary

Modifier and Type	Method and Description
(package private) static String	createValue(String row, String qual, long ts)
static Collection<Object[]>	generateParams(org.apache.hadoop.hbase.io.compress.Compression.Algorithm algo, boolean useDataBlockEncoding)
private static String	getRowQualStr(org.apache.hadoop.hbase.Cell kv)
private static boolean	matchesQuery(org.apache.hadoop.hbase.KeyValue kv, Set<String> qualSet, int maxVersions, Map<String,Long> lastDelTimeMap)
private static String	qualStr(org.apache.hadoop.hbase.KeyValue kv)
private static List<String>	sequentialStrings(String prefix, int n)
void	testMultiColumnScanner()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

private static final org.slf4j.Logger LOG

TABLE_NAME

private static final String TABLE_NAME

MAX_VERSIONS

static final int MAX_VERSIONS

See Also:

Constant Field Values

FAMILY

private static final String FAMILY

See Also:

Constant Field Values

FAMILY_BYTES

private static final byte[] FAMILY_BYTES

NUM_COLUMNS

private static final int NUM_COLUMNS

The size of the column qualifier set used. Increasing this parameter exponentially increases test time.

See Also:

Constant Field Values

MAX_COLUMN_BIT_MASK

private static final int MAX_COLUMN_BIT_MASK

See Also:

Constant Field Values

NUM_FLUSHES

private static final int NUM_FLUSHES

See Also:

Constant Field Values

NUM_ROWS

private static final int NUM_ROWS

See Also:

Constant Field Values

BIG_LONG

private static final long BIG_LONG

A large value of type long for use as a timestamp

See Also:

Constant Field Values

TIMESTAMPS

private static final long[] TIMESTAMPS

Timestamps to test with. Cannot use Long.MAX_VALUE here, because it will be replaced by an timestamp auto-generated based on the time.

COLUMN_SKIP_IN_STORE_FILE_PROB

private static final double COLUMN_SKIP_IN_STORE_FILE_PROB

The probability that a column is skipped in a store file.

See Also:

Constant Field Values

DELETE_PROBABILITY

private static final double DELETE_PROBABILITY

The probability to delete a row/column pair

See Also:

Constant Field Values

TEST_UTIL

private static final HBaseTestingUtility TEST_UTIL

comprAlgo

public org.apache.hadoop.hbase.io.compress.Compression.Algorithm comprAlgo

bloomType

public org.apache.hadoop.hbase.regionserver.BloomType bloomType

dataBlockEncoding

public org.apache.hadoop.hbase.io.encoding.DataBlockEncoding dataBlockEncoding

Constructor Detail

TestMultiColumnScanner

public TestMultiColumnScanner()

Method Detail

generateParams

public static Collection<Object[]> generateParams(org.apache.hadoop.hbase.io.compress.Compression.Algorithm algo,
                                                  boolean useDataBlockEncoding)

testMultiColumnScanner

public void testMultiColumnScanner()
                            throws IOException

Throws:

IOException

getRowQualStr

private static String getRowQualStr(org.apache.hadoop.hbase.Cell kv)

matchesQuery

private static boolean matchesQuery(org.apache.hadoop.hbase.KeyValue kv,
                                    Set<String> qualSet,
                                    int maxVersions,
                                    Map<String,Long> lastDelTimeMap)

qualStr

private static String qualStr(org.apache.hadoop.hbase.KeyValue kv)

createValue

static String createValue(String row,
                          String qual,
                          long ts)

sequentialStrings

private static List<String> sequentialStrings(String prefix,
                                              int n)