org.apache.hadoop.hbase.io.asyncfs

Class TestFanOutOneBlockAsyncDFSOutputHang

java.lang.Object

org.apache.hadoop.hbase.io.asyncfs.AsyncFSTestBase

org.apache.hadoop.hbase.io.asyncfs.TestFanOutOneBlockAsyncDFSOutputHang

public class TestFanOutOneBlockAsyncDFSOutputHang
extends AsyncFSTestBase

Testcase for HBASE-26679, here we introduce a separate test class and not put the testcase in TestFanOutOneBlockAsyncDFSOutput because we will send heartbeat to DN when there is no out going packet, the timeout is controlled by TestFanOutOneBlockAsyncDFSOutput.READ_TIMEOUT_MS,which is 2 seconds, it will keep sending package out and DN will respond immedately and then mess up the testing handler added by us. So in this test class we use the default value for timeout which is 60 seconds and it is enough for this test.

Field Summary

Fields

Modifier and Type	Field and Description
private static Class<? extends org.apache.hbase.thirdparty.io.netty.channel.Channel>	CHANNEL_CLASS
static HBaseClassTestRule	CLASS_RULE
private static org.apache.hbase.thirdparty.io.netty.channel.EventLoopGroup	EVENT_LOOP_GROUP
private static org.apache.hadoop.hdfs.DistributedFileSystem	FS
private static org.slf4j.Logger	LOG
private static org.apache.hadoop.hbase.io.asyncfs.monitor.StreamSlowMonitor	MONITOR
org.junit.rules.TestName	name
private static org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput	OUT

Fields inherited from class org.apache.hadoop.hbase.io.asyncfs.AsyncFSTestBase

CLUSTER, CLUSTER_TEST_DIR, UTIL

Constructor Summary

Constructors

Constructor and Description
TestFanOutOneBlockAsyncDFSOutputHang()

Method Summary

Modifier and Type	Method and Description
private static org.apache.hadoop.hdfs.MiniDFSCluster.DataNodeProperties	findAndKillFirstDataNode(org.apache.hadoop.hdfs.protocol.DatanodeInfo firstDatanodeInfo)
static void	setUp()
static void	tearDown()
void	testFlushHangWhenOneDataNodeFailedBeforeOtherDataNodeAck() This test is for HBASE-26679.

Methods inherited from class org.apache.hadoop.hbase.io.asyncfs.AsyncFSTestBase

setupClusterTestDir, shutdownMiniDFSCluster, startMiniDFSCluster

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

CLASS_RULE

public static final HBaseClassTestRule CLASS_RULE

LOG

private static final org.slf4j.Logger LOG

FS

private static org.apache.hadoop.hdfs.DistributedFileSystem FS

EVENT_LOOP_GROUP

private static org.apache.hbase.thirdparty.io.netty.channel.EventLoopGroup EVENT_LOOP_GROUP

CHANNEL_CLASS

private static Class<? extends org.apache.hbase.thirdparty.io.netty.channel.Channel> CHANNEL_CLASS

MONITOR

private static org.apache.hadoop.hbase.io.asyncfs.monitor.StreamSlowMonitor MONITOR

OUT

private static org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput OUT

name

public org.junit.rules.TestName name

Constructor Detail

TestFanOutOneBlockAsyncDFSOutputHang

public TestFanOutOneBlockAsyncDFSOutputHang()

Method Detail

setUp

public static void setUp()
                  throws Exception

Throws:

Exception

tearDown

public static void tearDown()
                     throws Exception

Throws:

Exception

testFlushHangWhenOneDataNodeFailedBeforeOtherDataNodeAck

public void testFlushHangWhenOneDataNodeFailedBeforeOtherDataNodeAck()
                                                              throws Exception

 This test is for HBASE-26679. Consider there are two dataNodes: dn1 and dn2,dn2 is a slow DN.
 The threads sequence before HBASE-26679 is:
 1.We write some data to FanOutOneBlockAsyncDFSOutput and then flush it, there are one
   FanOutOneBlockAsyncDFSOutput.Callback in
   FanOutOneBlockAsyncDFSOutput.waitingAckQueue.
 2.The ack from dn1 arrives firstly and triggers Netty to invoke
   FanOutOneBlockAsyncDFSOutput.completed(org.apache.hbase.thirdparty.io.netty.channel.Channel) with dn1's channel, then in
   FanOutOneBlockAsyncDFSOutput.completed(org.apache.hbase.thirdparty.io.netty.channel.Channel), dn1's channel is removed from
   FanOutOneBlockAsyncDFSOutput.Callback.unfinishedReplicas.
 3.But dn2 responds slowly, before dn2 sending ack,dn1 is shut down or have a exception,
   so FanOutOneBlockAsyncDFSOutput.failed(org.apache.hbase.thirdparty.io.netty.channel.Channel, java.util.function.Supplier<java.lang.Throwable>) is triggered by Netty with dn1's channel,
   and because the FanOutOneBlockAsyncDFSOutput.Callback.unfinishedReplicas does not
   contain dn1's channel,the FanOutOneBlockAsyncDFSOutput.Callback is skipped in
   FanOutOneBlockAsyncDFSOutput.failed(org.apache.hbase.thirdparty.io.netty.channel.Channel, java.util.function.Supplier<java.lang.Throwable>) method,and
   FanOutOneBlockAsyncDFSOutput.state is set to
   FanOutOneBlockAsyncDFSOutput.State.BROKEN,and dn1,dn2 are all closed at the end of
   FanOutOneBlockAsyncDFSOutput.failed(org.apache.hbase.thirdparty.io.netty.channel.Channel, java.util.function.Supplier<java.lang.Throwable>).
 4.FanOutOneBlockAsyncDFSOutput.failed(org.apache.hbase.thirdparty.io.netty.channel.Channel, java.util.function.Supplier<java.lang.Throwable>) is triggered again by dn2 because it is closed,
   but because FanOutOneBlockAsyncDFSOutput.state is already
   FanOutOneBlockAsyncDFSOutput.State.BROKEN,the whole
   FanOutOneBlockAsyncDFSOutput.failed(org.apache.hbase.thirdparty.io.netty.channel.Channel, java.util.function.Supplier<java.lang.Throwable>) is skipped. So wait on the future
   returned by FanOutOneBlockAsyncDFSOutput.flush(boolean) would be stuck for ever.
 After HBASE-26679, for above step 4,even if the FanOutOneBlockAsyncDFSOutput.state
 is already FanOutOneBlockAsyncDFSOutput.State.BROKEN, we would still try to trigger
 FanOutOneBlockAsyncDFSOutput.Callback.future.
 

Throws:

Exception

findAndKillFirstDataNode

private static org.apache.hadoop.hdfs.MiniDFSCluster.DataNodeProperties findAndKillFirstDataNode(org.apache.hadoop.hdfs.protocol.DatanodeInfo firstDatanodeInfo)