Package org.apache.hadoop.hbase.mapreduce

Class TestHFileOutputFormat2

java.lang.Object

org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2TestBase

org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat2

@Tag("org.apache.hadoop.hbase.testclassification.VerySlowMapReduceTests") @Tag("org.apache.hadoop.hbase.testclassification.LargeTests")
public class TestHFileOutputFormat2
extends HFileOutputFormat2TestBase

Simple test for HFileOutputFormat2. Sets up and runs a mapreduce job that writes hfile output. Creates a few inner classes to implement splits and an inputformat that emits keys and values.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
private static class	TestHFileOutputFormat2.ConfigurationCaptorConnection

Nested classes/interfaces inherited from class org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2TestBase

HFileOutputFormat2TestBase.RandomKVGeneratingMapper, HFileOutputFormat2TestBase.RandomPutGeneratingMapper

Field Summary

Fields

Modifier and Type	Field	Description
private static final org.slf4j.Logger	LOG

Fields inherited from class org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2TestBase

DEFAULT_VALUE_LENGTH, FAMILIES, FAMILY_NAME, ROWSPERSPLIT, TABLE_NAMES, UTIL

Constructor Summary

Constructors

Constructor	Description
TestHFileOutputFormat2()

Method Summary

Modifier and Type	Method	Description
private org.apache.hadoop.mapreduce.TaskAttemptContext	createTestTaskAttemptContext(org.apache.hadoop.mapreduce.Job job)
private Map<String,Integer>	getMockColumnFamiliesForBlockSize(int numCfs)
private Map<String,org.apache.hadoop.hbase.regionserver.BloomType>	getMockColumnFamiliesForBloomType(int numCfs)
private Map<String,org.apache.hadoop.hbase.io.compress.Compression.Algorithm>	getMockColumnFamiliesForCompression(int numCfs)
private Map<String,org.apache.hadoop.hbase.io.encoding.DataBlockEncoding>	getMockColumnFamiliesForDataBlockEncoding(int numCfs)
private String	getStoragePolicyName(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path)
private String	getStoragePolicyNameForOldHDFSVersion(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path)
static void	main(String[] args)
void	manualTest(String[] args)
private void	quickPoll(Callable<Boolean> c, int waitMs)
private void	setupMockColumnFamiliesForBlockSize(org.apache.hadoop.hbase.client.Table table, Map<String,Integer> familyToDataBlockEncoding)
private void	setupMockColumnFamiliesForBloomType(org.apache.hadoop.hbase.client.Table table, Map<String,org.apache.hadoop.hbase.regionserver.BloomType> familyToDataBlockEncoding)
private void	setupMockColumnFamiliesForCompression(org.apache.hadoop.hbase.client.Table table, Map<String,org.apache.hadoop.hbase.io.compress.Compression.Algorithm> familyToCompression)
private void	setupMockColumnFamiliesForDataBlockEncoding(org.apache.hadoop.hbase.client.Table table, Map<String,org.apache.hadoop.hbase.io.encoding.DataBlockEncoding> familyToDataBlockEncoding)
private void	setupMockStartKeys(org.apache.hadoop.hbase.client.RegionLocator table)
private void	setupMockTableName(org.apache.hadoop.hbase.client.RegionLocator table)
void	test_LATEST_TIMESTAMP_isReplaced()	Test that HFileOutputFormat2 RecordWriter amends timestamps if passed a keyvalue whose timestamp is HConstants.LATEST_TIMESTAMP.
void	test_TIMERANGE()
void	test_WritingTagData()	Test that HFileOutputFormat2 RecordWriter writes tags such as ttl into hfile.
void	testBlockStoragePolicy()
void	testColumnFamilySettings()	Test that HFileOutputFormat2 RecordWriter uses compression and bloom filter settings from the column family descriptor
void	TestConfigureCompression()
void	testExcludeAllFromMinorCompaction()	This test is to test the scenario happened in HBASE-6901.
void	testExcludeMinorCompaction()
void	testJobConfiguration()
void	testMRIncrementalLoadWithLocalityMultiCluster()
void	testSerializeDeserializeFamilyBlockSizeMap()	Test for HFileOutputFormat2.createFamilyBlockSizeMap(Configuration).
void	testSerializeDeserializeFamilyBloomTypeMap()	Test for HFileOutputFormat2.createFamilyBloomTypeMap(Configuration).
void	testSerializeDeserializeFamilyCompressionMap()	Test for HFileOutputFormat2.createFamilyCompressionMap(Configuration).
void	testSerializeDeserializeFamilyDataBlockEncodingMap()	Test for HFileOutputFormat2.createFamilyDataBlockEncodingMap(Configuration).
void	testWritingPEData()	Run small MR job.
private void	writeRandomKeyValues(org.apache.hadoop.mapreduce.RecordWriter<org.apache.hadoop.hbase.io.ImmutableBytesWritable,org.apache.hadoop.hbase.Cell> writer, org.apache.hadoop.mapreduce.TaskAttemptContext context, Set<byte[]> families, int numRows)	Write random values to the writer assuming a table created using HFileOutputFormat2TestBase.FAMILIES as column family descriptors

Methods inherited from class org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2TestBase

generateData, generateRandomSplitKeys, generateRandomStartKeys, runIncrementalPELoad, setupRandomGeneratorMapper

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

LOG

private static final org.slf4j.Logger LOG

Constructor Details

TestHFileOutputFormat2

public TestHFileOutputFormat2()

Method Details

test_LATEST_TIMESTAMP_isReplaced

@Disabled("Goes zombie too frequently; needs work. See HBASE-14563")
@Test
public void test_LATEST_TIMESTAMP_isReplaced()
                                      throws Exception

Test that HFileOutputFormat2 RecordWriter amends timestamps if passed a keyvalue whose timestamp is HConstants.LATEST_TIMESTAMP.

Throws:

Exception

See Also:

• HBASE-2615

createTestTaskAttemptContext

private org.apache.hadoop.mapreduce.TaskAttemptContext createTestTaskAttemptContext(org.apache.hadoop.mapreduce.Job job)
                                                                             throws Exception

Throws:

Exception

test_TIMERANGE

@Disabled("Goes zombie too frequently; needs work. See HBASE-14563")
@Test
public void test_TIMERANGE()
                    throws Exception

Throws:

Exception

testWritingPEData

@Disabled("Goes zombie too frequently; needs work. See HBASE-14563")
@Test
public void testWritingPEData()
                       throws Exception

Run small MR job.

Throws:

Exception

test_WritingTagData

@Test
public void test_WritingTagData()
                         throws Exception

Test that HFileOutputFormat2 RecordWriter writes tags such as ttl into hfile.

Throws:

Exception

testJobConfiguration

@Disabled("Goes zombie too frequently; needs work. See HBASE-14563")
@Test
public void testJobConfiguration()
                          throws Exception

Throws:

Exception

testSerializeDeserializeFamilyCompressionMap

@Disabled("Goes zombie too frequently; needs work. See HBASE-14563")
@Test
public void testSerializeDeserializeFamilyCompressionMap()
                                                  throws IOException

Test for HFileOutputFormat2.createFamilyCompressionMap(Configuration). Tests that the family compression map is correctly serialized into and deserialized from configuration

Throws:

IOException

setupMockColumnFamiliesForCompression

private void setupMockColumnFamiliesForCompression(org.apache.hadoop.hbase.client.Table table,
 Map<String,org.apache.hadoop.hbase.io.compress.Compression.Algorithm> familyToCompression)
                                            throws IOException

Throws:

IOException

getMockColumnFamiliesForCompression

private Map<String,org.apache.hadoop.hbase.io.compress.Compression.Algorithm> getMockColumnFamiliesForCompression(int numCfs)

Returns:

a map from column family names to compression algorithms for testing column family compression. Column family names have special characters

testSerializeDeserializeFamilyBloomTypeMap

@Disabled("Goes zombie too frequently; needs work. See HBASE-14563")
@Test
public void testSerializeDeserializeFamilyBloomTypeMap()
                                                throws IOException

Test for HFileOutputFormat2.createFamilyBloomTypeMap(Configuration). Tests that the family bloom type map is correctly serialized into and deserialized from configuration

Throws:

IOException

setupMockColumnFamiliesForBloomType

private void setupMockColumnFamiliesForBloomType(org.apache.hadoop.hbase.client.Table table,
 Map<String,org.apache.hadoop.hbase.regionserver.BloomType> familyToDataBlockEncoding)
                                          throws IOException

Throws:

IOException

getMockColumnFamiliesForBloomType

private Map<String,org.apache.hadoop.hbase.regionserver.BloomType> getMockColumnFamiliesForBloomType(int numCfs)

Returns:

a map from column family names to compression algorithms for testing column family compression. Column family names have special characters

testSerializeDeserializeFamilyBlockSizeMap

@Disabled("Goes zombie too frequently; needs work. See HBASE-14563")
@Test
public void testSerializeDeserializeFamilyBlockSizeMap()
                                                throws IOException

Test for HFileOutputFormat2.createFamilyBlockSizeMap(Configuration). Tests that the family block size map is correctly serialized into and deserialized from configuration

Throws:

IOException

setupMockColumnFamiliesForBlockSize

private void setupMockColumnFamiliesForBlockSize(org.apache.hadoop.hbase.client.Table table,
 Map<String,Integer> familyToDataBlockEncoding)
                                          throws IOException

Throws:

IOException

getMockColumnFamiliesForBlockSize

private Map<String,Integer> getMockColumnFamiliesForBlockSize(int numCfs)

Returns:

a map from column family names to compression algorithms for testing column family compression. Column family names have special characters

testSerializeDeserializeFamilyDataBlockEncodingMap

@Disabled("Goes zombie too frequently; needs work. See HBASE-14563")
@Test
public void testSerializeDeserializeFamilyDataBlockEncodingMap()
                                                        throws IOException

Test for HFileOutputFormat2.createFamilyDataBlockEncodingMap(Configuration). Tests that the family data block encoding map is correctly serialized into and deserialized from configuration

Throws:

IOException

setupMockColumnFamiliesForDataBlockEncoding

private void setupMockColumnFamiliesForDataBlockEncoding(org.apache.hadoop.hbase.client.Table table,
 Map<String,org.apache.hadoop.hbase.io.encoding.DataBlockEncoding> familyToDataBlockEncoding)
                                                  throws IOException

Throws:

IOException

getMockColumnFamiliesForDataBlockEncoding

private Map<String,org.apache.hadoop.hbase.io.encoding.DataBlockEncoding> getMockColumnFamiliesForDataBlockEncoding(int numCfs)

Returns:

a map from column family names to compression algorithms for testing column family compression. Column family names have special characters

setupMockStartKeys

private void setupMockStartKeys(org.apache.hadoop.hbase.client.RegionLocator table)
                         throws IOException

Throws:

IOException

setupMockTableName

private void setupMockTableName(org.apache.hadoop.hbase.client.RegionLocator table)
                         throws IOException

Throws:

IOException

testColumnFamilySettings

@Disabled("Goes zombie too frequently; needs work. See HBASE-14563")
@Test
public void testColumnFamilySettings()
                              throws Exception

Test that HFileOutputFormat2 RecordWriter uses compression and bloom filter settings from the column family descriptor

Throws:

Exception

writeRandomKeyValues

private void writeRandomKeyValues(org.apache.hadoop.mapreduce.RecordWriter<org.apache.hadoop.hbase.io.ImmutableBytesWritable,org.apache.hadoop.hbase.Cell> writer,
 org.apache.hadoop.mapreduce.TaskAttemptContext context,
 Set<byte[]> families,
 int numRows)
                           throws IOException,
InterruptedException

Write random values to the writer assuming a table created using HFileOutputFormat2TestBase.FAMILIES as column family descriptors

Throws:

IOException

InterruptedException

testExcludeAllFromMinorCompaction

@Disabled("Flakey: See HBASE-9051")
@Test
public void testExcludeAllFromMinorCompaction()
                                       throws Exception

This test is to test the scenario happened in HBASE-6901. All files are bulk loaded and excluded from minor compaction. Without the fix of HBASE-6901, an ArrayIndexOutOfBoundsException will be thrown.

Throws:

Exception

testExcludeMinorCompaction

@Disabled("Goes zombie too frequently; needs work. See HBASE-14563")
@Test
public void testExcludeMinorCompaction()
                                throws Exception

Throws:

Exception

quickPoll

private void quickPoll(Callable<Boolean> c,
 int waitMs)
                throws Exception

Throws:

Exception

main

public static void main(String[] args)
                 throws Exception

Throws:

Exception

manualTest

public void manualTest(String[] args)
                throws Exception

Throws:

Exception

testBlockStoragePolicy

@Test
public void testBlockStoragePolicy()
                            throws Exception

Throws:

Exception

getStoragePolicyName

private String getStoragePolicyName(org.apache.hadoop.fs.FileSystem fs,
 org.apache.hadoop.fs.Path path)

getStoragePolicyNameForOldHDFSVersion

private String getStoragePolicyNameForOldHDFSVersion(org.apache.hadoop.fs.FileSystem fs,
 org.apache.hadoop.fs.Path path)

TestConfigureCompression

@Test
public void TestConfigureCompression()
                              throws Exception

Throws:

Exception

testMRIncrementalLoadWithLocalityMultiCluster

@Test
public void testMRIncrementalLoadWithLocalityMultiCluster()
                                                   throws Exception

Throws:

Exception