Package org.apache.hadoop.hbase.mapreduce

Class TestWALPlayer

java.lang.Object

org.apache.hadoop.hbase.mapreduce.TestWALPlayer

@Tag("org.apache.hadoop.hbase.testclassification.MapReduceTests") @Tag("org.apache.hadoop.hbase.testclassification.LargeTests")
public class TestWALPlayer
extends Object

Basic test for the WALPlayer M/R tool

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
private static interface	TestWALPlayer.TestMethod

Field Summary

Fields

Modifier and Type	Field	Description
private static String	bulkLoadOutputDir
private static SingleProcessHBaseCluster	cluster
private static final byte[]	COLUMN1
private static final byte[]	COLUMN2
private static org.apache.hadoop.conf.Configuration	conf
private static final byte[]	FAMILY
private static org.apache.hadoop.fs.FileSystem	hdfs
private static org.apache.hadoop.fs.FileSystem	localFs
private static org.apache.hadoop.fs.FileSystem	logFs
private static org.apache.hadoop.fs.Path	rootDir
private static final byte[]	ROW
private static final HBaseTestingUtil	TEST_UTIL
private static org.apache.hadoop.fs.Path	walRootDir

Constructor Summary

Constructors

Constructor	Description
TestWALPlayer()

Method Summary

Modifier and Type	Method	Description
static void	afterClass()
private void	assertMultiTableOutputFormatDirStructure(org.apache.hadoop.hbase.TableName tableName, String namespace)
static void	beforeClass()
private org.apache.hadoop.fs.Path	createEmptyWALFile(String walDir)
private void	putRowIntoTable(org.apache.hadoop.hbase.client.Table table)
private static void	runWithDiskBasedSortingDisabledAndEnabled(TestWALPlayer.TestMethod method)
private void	setConfSimilarToIncrementalBackupWALToHFilesMethod(org.junit.jupiter.api.TestInfo testInfo, org.apache.hadoop.conf.Configuration conf)
void	testIgnoreEmptyWALFiles()
void	testMainMethod()	Test main method
void	testPlayingRecoveredEdit()	Test that WALPlayer can replay recovered.edits files.
void	testWALKeyValueMapper()	Test WALKeyValueMapper setup and map
private void	testWALKeyValueMapper(String tableConfigKey)
void	testWALKeyValueMapperWithDeprecatedConfig()
void	testWALPlayer(org.junit.jupiter.api.TestInfo testInfo)	Simple end-to-end test
void	testWALPlayerBulkLoadWithOverriddenTimestamps(org.junit.jupiter.api.TestInfo testInfo)	Tests that when you write multiple cells with the same timestamp they are properly sorted by their sequenceId in WALPlayer/CellSortReducer so that the correct one wins when querying from the resulting bulkloaded HFiles.
void	testWALPlayerMultiTableHFileOutputFormat(org.junit.jupiter.api.TestInfo testInfo)	Verifies the HFile output format for WALPlayer has the following directory structure when "hbase.mapreduce.use.multi.table.hfileoutputformat" is set to true: .../BULK_OUTPUT_CONF_KEY/namespace/tableName/columnFamily
void	testWALPlayerSingleTableHFileOutputFormat(org.junit.jupiter.api.TestInfo testInfo)	Verifies the HFile output format for WALPlayer has the following directory structure when "hbase.mapreduce.use.multi.table.hfileoutputformat" is set to false: .../BULK_OUTPUT_CONF_KEY/columnFamily Also verifies an exception occurs when the WALPlayer is run on multiple tables at once while hbase.mapreduce.use.multi.table.hfileoutputformat is set to false.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

FAMILY

private static final byte[] FAMILY

COLUMN1

private static final byte[] COLUMN1

COLUMN2

private static final byte[] COLUMN2

ROW

private static final byte[] ROW

TEST_UTIL

private static final HBaseTestingUtil TEST_UTIL

cluster

private static SingleProcessHBaseCluster cluster

rootDir

private static org.apache.hadoop.fs.Path rootDir

walRootDir

private static org.apache.hadoop.fs.Path walRootDir

localFs

private static org.apache.hadoop.fs.FileSystem localFs

logFs

private static org.apache.hadoop.fs.FileSystem logFs

conf

private static org.apache.hadoop.conf.Configuration conf

hdfs

private static org.apache.hadoop.fs.FileSystem hdfs

bulkLoadOutputDir

private static String bulkLoadOutputDir

Constructor Details

TestWALPlayer

public TestWALPlayer()

Method Details

beforeClass

@BeforeAll
public static void beforeClass()
                        throws Exception

Throws:

Exception

afterClass

@AfterAll
public static void afterClass()
                       throws Exception

Throws:

Exception

testPlayingRecoveredEdit

@Test
public void testPlayingRecoveredEdit()
                              throws Exception

Test that WALPlayer can replay recovered.edits files.

Throws:

Exception

testWALPlayerBulkLoadWithOverriddenTimestamps

@Test
public void testWALPlayerBulkLoadWithOverriddenTimestamps(org.junit.jupiter.api.TestInfo testInfo)
                                                   throws Exception

Tests that when you write multiple cells with the same timestamp they are properly sorted by their sequenceId in WALPlayer/CellSortReducer so that the correct one wins when querying from the resulting bulkloaded HFiles. See HBASE-27649

Throws:

Exception

testWALPlayer

@Test
public void testWALPlayer(org.junit.jupiter.api.TestInfo testInfo)
                   throws Exception

Simple end-to-end test

Throws:

Exception

testWALKeyValueMapper

@Test
public void testWALKeyValueMapper()
                           throws Exception

Test WALKeyValueMapper setup and map

Throws:

Exception

testWALKeyValueMapperWithDeprecatedConfig

@Test
public void testWALKeyValueMapperWithDeprecatedConfig()
                                               throws Exception

Throws:

Exception

testWALKeyValueMapper

private void testWALKeyValueMapper(String tableConfigKey)
                            throws Exception

Throws:

Exception

testMainMethod

@Test
public void testMainMethod()
                    throws Exception

Test main method

Throws:

Exception

runWithDiskBasedSortingDisabledAndEnabled

private static void runWithDiskBasedSortingDisabledAndEnabled(TestWALPlayer.TestMethod method)
                                                       throws Exception

Throws:

Exception

testIgnoreEmptyWALFiles

@Test
public void testIgnoreEmptyWALFiles()
                             throws Exception

Throws:

Exception

testWALPlayerMultiTableHFileOutputFormat

@Test
public void testWALPlayerMultiTableHFileOutputFormat(org.junit.jupiter.api.TestInfo testInfo)
                                              throws Exception

Verifies the HFile output format for WALPlayer has the following directory structure when "hbase.mapreduce.use.multi.table.hfileoutputformat" is set to true:

.../BULK_OUTPUT_CONF_KEY/namespace/tableName/columnFamily

Throws:

Exception

testWALPlayerSingleTableHFileOutputFormat

@Test
public void testWALPlayerSingleTableHFileOutputFormat(org.junit.jupiter.api.TestInfo testInfo)
                                               throws Exception

Verifies the HFile output format for WALPlayer has the following directory structure when "hbase.mapreduce.use.multi.table.hfileoutputformat" is set to false:

.../BULK_OUTPUT_CONF_KEY/columnFamily

Also verifies an exception occurs when the WALPlayer is run on multiple tables at once while hbase.mapreduce.use.multi.table.hfileoutputformat is set to false.

Throws:

Exception

putRowIntoTable

private void putRowIntoTable(org.apache.hadoop.hbase.client.Table table)
                      throws IOException

Throws:

IOException

createEmptyWALFile

private org.apache.hadoop.fs.Path createEmptyWALFile(String walDir)
                                              throws IOException

Throws:

IOException

setConfSimilarToIncrementalBackupWALToHFilesMethod

private void setConfSimilarToIncrementalBackupWALToHFilesMethod(org.junit.jupiter.api.TestInfo testInfo,
 org.apache.hadoop.conf.Configuration conf)

assertMultiTableOutputFormatDirStructure

private void assertMultiTableOutputFormatDirStructure(org.apache.hadoop.hbase.TableName tableName,
 String namespace)
                                               throws IOException

Throws:

IOException