Package org.apache.hadoop.hbase.regionserver

Class TestCompaction

java.lang.Object

org.apache.hadoop.hbase.regionserver.TestCompaction

Direct Known Subclasses:

TestCompactionWithCoprocessor

@Tag("org.apache.hadoop.hbase.testclassification.RegionServerTests") @Tag("org.apache.hadoop.hbase.testclassification.LargeTests")
public class TestCompaction
extends Object

Test compaction framework and common functions

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
class	TestCompaction.BlockingStoreMockMaker
static class	TestCompaction.DummyCompactor
(package private) class	TestCompaction.StoreMockMaker
static class	TestCompaction.Tracker	Simple CompactionLifeCycleTracker on which you can wait until the requested compaction finishes.
static class	TestCompaction.WaitThroughPutController	Simple CompactionLifeCycleTracker on which you can wait until the requested compaction finishes.

Field Summary

Fields

Modifier and Type	Field	Description
private static final byte[]	COLUMN_FAMILY
private static final byte[]	COLUMN_FAMILY_TEXT
private int	compactionThreshold
protected org.apache.hadoop.conf.Configuration	conf
private final byte[]	FAMILY
private static final long	MAX_FILES_TO_COMPACT
private String	name
private org.apache.hadoop.hbase.regionserver.HRegion	r
private byte[]	secondRowBytes
private final byte[]	STARTROW
private org.apache.hadoop.hbase.client.TableDescriptor	tableDescriptor
private byte[]	thirdRowBytes
private static final HBaseTestingUtil	UTIL

Constructor Summary

Constructors

Constructor	Description
TestCompaction()	constructor

Method Summary

Modifier and Type	Method	Description
private int	count()
private static org.apache.hadoop.hbase.regionserver.HStoreFile	createFile()
private void	createStoreFile(org.apache.hadoop.hbase.regionserver.HRegion region)
private void	createStoreFile(org.apache.hadoop.hbase.regionserver.HRegion region, String family)
void	generateHFileForCorruptBlockTest()	Generates the HFile used by testCompactionWithCorruptBlock().
void	setUp(org.junit.jupiter.api.TestInfo testInfo)
void	tearDown()
void	testCompactionFailure()
void	testCompactionQueuePriorities()	Test compaction priority management and multiple compactions per store (HBASE-8665).
void	testCompactionSeqId()	Firstly write 10 cells (with different time stamp) to a qualifier and flush to hfile1, then write 10 cells (with different time stamp) to the same qualifier and flush to hfile2.
void	testCompactionWithCorruptBlock()	This test uses a hand-modified HFile, which is loaded in from the resources' path.
void	testCompactionWithCorruptResult()
void	testInterruptCompactionBySize()	Verify that you can stop a long-running compaction (used during RS shutdown)
void	testInterruptCompactionByTime()
void	testInterruptingRunningCompactions()
void	testMultipleCustomCompactionRequests()	HBASE-7947: Regression test to ensure adding to the correct list in the CompactSplit
void	testStopStartCompaction()	Test no new Compaction requests are generated after calling stop compactions
void	testTrackingCompactionRequest()	Create a custom compaction request and be sure that we can track it through the queue, knowing when the compaction is completed.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

name

private String name

UTIL

private static final HBaseTestingUtil UTIL

conf

protected org.apache.hadoop.conf.Configuration conf

r

private org.apache.hadoop.hbase.regionserver.HRegion r

tableDescriptor

private org.apache.hadoop.hbase.client.TableDescriptor tableDescriptor

COLUMN_FAMILY

private static final byte[] COLUMN_FAMILY

STARTROW

private final byte[] STARTROW

COLUMN_FAMILY_TEXT

private static final byte[] COLUMN_FAMILY_TEXT

compactionThreshold

private int compactionThreshold

secondRowBytes

private byte[] secondRowBytes

thirdRowBytes

private byte[] thirdRowBytes

MAX_FILES_TO_COMPACT

private static final long MAX_FILES_TO_COMPACT

See Also:

• Constant Field Values

FAMILY

private final byte[] FAMILY

Constructor Details

TestCompaction

public TestCompaction()

constructor

Method Details

setUp

@BeforeEach
public void setUp(org.junit.jupiter.api.TestInfo testInfo)
           throws Exception

Throws:

Exception

tearDown

@AfterEach
public void tearDown()
              throws Exception

Throws:

Exception

testInterruptCompactionBySize

@Test
public void testInterruptCompactionBySize()
                                   throws Exception

Verify that you can stop a long-running compaction (used during RS shutdown)

Throws:

Exception

testInterruptCompactionByTime

@Test
public void testInterruptCompactionByTime()
                                   throws Exception

Throws:

Exception

count

private int count()
           throws IOException

Throws:

IOException

createStoreFile

private void createStoreFile(org.apache.hadoop.hbase.regionserver.HRegion region)
                      throws IOException

Throws:

IOException

createStoreFile

private void createStoreFile(org.apache.hadoop.hbase.regionserver.HRegion region,
 String family)
                      throws IOException

Throws:

IOException

testCompactionWithCorruptResult

@Test
public void testCompactionWithCorruptResult()
                                     throws Exception

Throws:

Exception

generateHFileForCorruptBlockTest

@Disabled("Not a test; utility for regenerating testCompactionWithCorruptBlock resource file")
@Test
public void generateHFileForCorruptBlockTest()
                                      throws Exception

Generates the HFile used by testCompactionWithCorruptBlock(). Run this method to regenerate the test resource file after changes to the HFile format. The output file must then be hand-edited to corrupt the first data block (zero out the GZip magic bytes at offset 33) before being placed into the test resources directory.

Throws:

Exception

testCompactionWithCorruptBlock

@Test
public void testCompactionWithCorruptBlock()
                                    throws Exception

This test uses a hand-modified HFile, which is loaded in from the resources' path. That file was generated from generateHFileForCorruptBlockTest() and then edited to corrupt the GZ-encoded block by zeroing-out the first two bytes of the GZip header, the "standard declaration" of 1f 8b, found at offset 33 in the file. I'm not sure why, but it seems that in this test context we do not enforce CRC checksums. Thus, this corruption manifests in the Decompressor rather than in the reader when it loads the block bytes and compares vs. the header.

Throws:

Exception

testTrackingCompactionRequest

@Test
public void testTrackingCompactionRequest()
                                   throws Exception

Create a custom compaction request and be sure that we can track it through the queue, knowing when the compaction is completed.

Throws:

Exception

testCompactionFailure

@Test
public void testCompactionFailure()
                           throws Exception

Throws:

Exception

testStopStartCompaction

@Test
public void testStopStartCompaction()
                             throws IOException

Test no new Compaction requests are generated after calling stop compactions

Throws:

IOException

testInterruptingRunningCompactions

@Test
public void testInterruptingRunningCompactions()
                                        throws Exception

Throws:

Exception

testMultipleCustomCompactionRequests

@Test
public void testMultipleCustomCompactionRequests()
                                          throws Exception

HBASE-7947: Regression test to ensure adding to the correct list in the CompactSplit

Throws:

Exception - on failure

testCompactionQueuePriorities

@Test
public void testCompactionQueuePriorities()
                                   throws Exception

Test compaction priority management and multiple compactions per store (HBASE-8665).

Throws:

Exception

testCompactionSeqId

@Test
public void testCompactionSeqId()
                         throws Exception

Firstly write 10 cells (with different time stamp) to a qualifier and flush to hfile1, then write 10 cells (with different time stamp) to the same qualifier and flush to hfile2. The latest cell (cell-A) in hfile1 and the oldest cell (cell-B) in hfile2 are with the same time stamp but different sequence id, and will get scanned successively during compaction.

We set compaction.kv.max to 10 so compaction will scan 10 versions each round, meanwhile we set keepSeqIdPeriod=0 in TestCompaction.DummyCompactor so all 10 versions of hfile2 will be written out with seqId cleaned (set to 0) including cell-B, then when scanner goes to cell-A it will cause a scan out-of-order assertion error before HBASE-16931 if error occurs during the test

Throws:

Exception

createFile

private static org.apache.hadoop.hbase.regionserver.HStoreFile createFile()
                                                                   throws Exception

Throws:

Exception