Package org.apache.hadoop.hbase.replication.regionserver

Class TestBasicWALEntryStream

java.lang.Object

org.apache.hadoop.hbase.replication.regionserver.WALEntryStreamTestBase

org.apache.hadoop.hbase.replication.regionserver.TestBasicWALEntryStream

Direct Known Subclasses:

TestBasicWALEntryStreamAsyncFSWAL, TestBasicWALEntryStreamFSHLog

public abstract class TestBasicWALEntryStream
extends WALEntryStreamTestBase

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	TestBasicWALEntryStream.FailingWALEntryFilter
private static class	TestBasicWALEntryStream.PartialWALEntryFailingWALEntryFilter

Nested classes/interfaces inherited from class org.apache.hadoop.hbase.replication.regionserver.WALEntryStreamTestBase

WALEntryStreamTestBase.PathWatcher, WALEntryStreamTestBase.WALEntryStreamWithRetries

Field Summary

Fields

Modifier and Type	Field	Description
boolean	isCompressionEnabled

Fields inherited from class org.apache.hadoop.hbase.replication.regionserver.WALEntryStreamTestBase

cluster, CONF, fakeWalGroupId, family, fs, info, log, logQueue, mvcc, pathWatcher, qualifier, scopes, tableName, TEST_TIMEOUT_MS, TEST_UTIL, tn

Constructor Summary

Constructors

Constructor	Description
TestBasicWALEntryStream()

Method Summary

Modifier and Type	Method	Description
private void	appendEntries(org.apache.hadoop.hbase.wal.WALProvider.Writer writer, int numEntries)
private void	appendEntriesToLogAndSync(int count)
private void	appendToLog(String key)
private org.apache.hadoop.hbase.replication.regionserver.MetricsReplicationGlobalSourceSource	createMockGlobalMetrics()
private org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader	createReader(boolean recovered, org.apache.hadoop.conf.Configuration conf)
private org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader	createReaderWithBadReplicationFilter(int numFailures, org.apache.hadoop.conf.Configuration conf)
static Iterable<Object[]>	data()
private org.apache.hadoop.hbase.replication.WALEntryFilter	getDummyFilter()
private org.apache.hadoop.hbase.replication.WALEntryFilter	getIntermittentFailingFilter(int numFailuresInFilter)
private PriorityBlockingQueue<org.apache.hadoop.fs.Path>	getQueue()
private String	getRow(org.apache.hadoop.hbase.wal.WAL.Entry entry)
private org.apache.hadoop.hbase.wal.WALEdit	getWALEdit(String row)
private org.apache.hadoop.hbase.replication.regionserver.ReplicationSource	mockReplicationSource(boolean recovered, org.apache.hadoop.conf.Configuration conf)
private org.apache.hadoop.hbase.wal.WAL.Entry	next(org.apache.hadoop.hbase.replication.regionserver.WALEntryStream entryStream)
void	setUp()
void	testAppendsWithRolls()	Tests basic reading of log appends
void	testCleanClosedWALs()	Tests that wals are closed cleanly and we read the trailer when we remove wal from WALEntryStream.
void	testEmptyStream()
void	testEOFExceptionForRecoveredQueue()	Test removal of 0 length log from logQueue if the source is a recovered source and size of logQueue is only 1.
void	testEOFExceptionForRecoveredQueueWithMultipleLogs()
void	testEOFExceptionInOldWALsDirectory()	Tests that we handle EOFException properly if the wal has moved to oldWALs directory.
void	testLogRollWhileStreaming()	Tests that if after a stream is opened, more entries come in and then the log is rolled, we don't mistakenly dequeue the current log thinking we're done with it
void	testNewEntriesWhileStreaming()	Tests that if writes come in while we have a stream open, we shouldn't miss them
void	testPosition()	Tests that if we stop before hitting the end of a stream, we can continue where we left off using the last position
void	testReadBeyondCommittedLength()
void	testReplicationSourceWALReader()
void	testReplicationSourceWALReaderDisabled()
void	testReplicationSourceWALReaderRecovered()
void	testReplicationSourceWALReaderWithFailingFilter()
void	testReplicationSourceWALReaderWithPartialWALEntryFailingFilter()	This test is for HBASE-27778, when WALEntryFilter.filter(org.apache.hadoop.hbase.wal.WAL.Entry) throws exception for some entries in WALEntryBatch,ReplicationSourceWALReader#totalBufferUsed should be decreased because WALEntryBatch is not put to ReplicationSourceWALReader.entryBatchQueue.
void	testReplicationSourceWALReaderWrongPosition()
void	testResumeStreamingFromPosition()
void	testSizeOfLogQueue()	Tests size of log queue is incremented and decremented properly.
void	testWALEntryStreamEOFRightAfterHeader()
void	testWALKeySerialization()

Methods inherited from class org.apache.hadoop.hbase.replication.regionserver.WALEntryStreamTestBase

appendToLog, appendToLogAndSync, appendToLogAndSync, getWALEdits, initWAL, startCluster, tearDown, tearDownAfterClass

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

isCompressionEnabled

public boolean isCompressionEnabled

Constructor Details

TestBasicWALEntryStream

public TestBasicWALEntryStream()

Method Details

data

public static Iterable<Object[]> data()

setUp

public void setUp()
           throws Exception

Throws:

Exception

next

private org.apache.hadoop.hbase.wal.WAL.Entry next(org.apache.hadoop.hbase.replication.regionserver.WALEntryStream entryStream)

testAppendsWithRolls

public void testAppendsWithRolls()
                          throws Exception

Tests basic reading of log appends

Throws:

Exception

testLogRollWhileStreaming

public void testLogRollWhileStreaming()
                               throws Exception

Tests that if after a stream is opened, more entries come in and then the log is rolled, we don't mistakenly dequeue the current log thinking we're done with it

Throws:

Exception

testNewEntriesWhileStreaming

public void testNewEntriesWhileStreaming()
                                  throws Exception

Tests that if writes come in while we have a stream open, we shouldn't miss them

Throws:

Exception

testResumeStreamingFromPosition

public void testResumeStreamingFromPosition()
                                     throws Exception

Throws:

Exception

testPosition

public void testPosition()
                  throws Exception

Tests that if we stop before hitting the end of a stream, we can continue where we left off using the last position

Throws:

Exception

testEmptyStream

public void testEmptyStream()
                     throws Exception

Throws:

Exception

testWALKeySerialization

public void testWALKeySerialization()
                             throws Exception

Throws:

Exception

mockReplicationSource

private org.apache.hadoop.hbase.replication.regionserver.ReplicationSource mockReplicationSource(boolean recovered,
 org.apache.hadoop.conf.Configuration conf)
                                                                                          throws IOException

Throws:

IOException

createMockGlobalMetrics

private org.apache.hadoop.hbase.replication.regionserver.MetricsReplicationGlobalSourceSource createMockGlobalMetrics()

createReader

private org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader createReader(boolean recovered,
 org.apache.hadoop.conf.Configuration conf)
                                                                                          throws IOException

Throws:

IOException

createReaderWithBadReplicationFilter

private org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader createReaderWithBadReplicationFilter(int numFailures,
 org.apache.hadoop.conf.Configuration conf)
                                                                                                                  throws IOException

Throws:

IOException

testReplicationSourceWALReader

public void testReplicationSourceWALReader()
                                    throws Exception

Throws:

Exception

testReplicationSourceWALReaderWithFailingFilter

public void testReplicationSourceWALReaderWithFailingFilter()
                                                     throws Exception

Throws:

Exception

testReplicationSourceWALReaderRecovered

public void testReplicationSourceWALReaderRecovered()
                                             throws Exception

Throws:

Exception

testReplicationSourceWALReaderWrongPosition

public void testReplicationSourceWALReaderWrongPosition()
                                                 throws Exception

Throws:

Exception

testReplicationSourceWALReaderDisabled

public void testReplicationSourceWALReaderDisabled()
                                            throws IOException,
InterruptedException,
ExecutionException

Throws:

IOException

InterruptedException

ExecutionException

getRow

private String getRow(org.apache.hadoop.hbase.wal.WAL.Entry entry)

appendToLog

private void appendToLog(String key)
                  throws IOException

Throws:

IOException

appendEntriesToLogAndSync

private void appendEntriesToLogAndSync(int count)
                                throws IOException

Throws:

IOException

getWALEdit

private org.apache.hadoop.hbase.wal.WALEdit getWALEdit(String row)

getDummyFilter

private org.apache.hadoop.hbase.replication.WALEntryFilter getDummyFilter()

getIntermittentFailingFilter

private org.apache.hadoop.hbase.replication.WALEntryFilter getIntermittentFailingFilter(int numFailuresInFilter)

testReadBeyondCommittedLength

public void testReadBeyondCommittedLength()
                                   throws IOException,
InterruptedException

Throws:

IOException

InterruptedException

testEOFExceptionForRecoveredQueue

public void testEOFExceptionForRecoveredQueue()
                                       throws Exception

Test removal of 0 length log from logQueue if the source is a recovered source and size of logQueue is only 1.

Throws:

Exception

testEOFExceptionForRecoveredQueueWithMultipleLogs

public void testEOFExceptionForRecoveredQueueWithMultipleLogs()
                                                       throws Exception

Throws:

Exception

getQueue

private PriorityBlockingQueue<org.apache.hadoop.fs.Path> getQueue()

appendEntries

private void appendEntries(org.apache.hadoop.hbase.wal.WALProvider.Writer writer,
 int numEntries)
                    throws IOException

Throws:

IOException

testSizeOfLogQueue

public void testSizeOfLogQueue()
                        throws Exception

Tests size of log queue is incremented and decremented properly.

Throws:

Exception

testCleanClosedWALs

public void testCleanClosedWALs()
                         throws Exception

Tests that wals are closed cleanly and we read the trailer when we remove wal from WALEntryStream.

Throws:

Exception

testEOFExceptionInOldWALsDirectory

public void testEOFExceptionInOldWALsDirectory()
                                        throws Exception

Tests that we handle EOFException properly if the wal has moved to oldWALs directory.

Throws:

Exception

testReplicationSourceWALReaderWithPartialWALEntryFailingFilter

public void testReplicationSourceWALReaderWithPartialWALEntryFailingFilter()
                                                                    throws Exception

This test is for HBASE-27778, when WALEntryFilter.filter(org.apache.hadoop.hbase.wal.WAL.Entry) throws exception for some entries in WALEntryBatch,ReplicationSourceWALReader#totalBufferUsed should be decreased because WALEntryBatch is not put to ReplicationSourceWALReader.entryBatchQueue.

Throws:

Exception

testWALEntryStreamEOFRightAfterHeader

public void testWALEntryStreamEOFRightAfterHeader()
                                           throws Exception

Throws:

Exception