Class StreamSlowMonitor
java.lang.Object
org.apache.hadoop.hbase.io.asyncfs.monitor.StreamSlowMonitor
- All Implemented Interfaces:
ConfigurationObserver
Class for monitor the wal file flush performance. Each active wal file has a StreamSlowMonitor.
-
Nested Class Summary
-
Field Summary
Modifier and TypeFieldDescriptionprivate static final String
Configure for the speed check of packet min length.private static final String
Configure for the check of large packet(which is configured byDATANODE_PACKET_FLUSH_CHECK_SPEED_MIN_DATA_LENGTH_KEY
) flush speed.static final String
Configure for the slow packet process time, a duration from send to ACK.private final org.apache.hbase.thirdparty.com.google.common.cache.LoadingCache<org.apache.hadoop.hdfs.protocol.DatanodeInfo,
Deque<StreamSlowMonitor.PacketAckData>> private static final long
private static final double
private static final long
private static final long
private static final int
private final ExcludeDatanodeManager
private static final org.slf4j.Logger
private long
private double
private int
private final String
private long
private long
private static final String
Configure for the TTL of the data that a datanode detected slow.private static final String
Configure for the min count for a datanode detected slow. -
Constructor Summary
ConstructorDescriptionStreamSlowMonitor
(org.apache.hadoop.conf.Configuration conf, String name, ExcludeDatanodeManager excludeDatanodeManager) -
Method Summary
Modifier and TypeMethodDescriptionprivate boolean
addSlowAckData
(org.apache.hadoop.hdfs.protocol.DatanodeInfo datanodeInfo, long dataLength, long processTime) void
checkProcessTimeAndSpeed
(org.apache.hadoop.hdfs.protocol.DatanodeInfo datanodeInfo, long packetDataLen, long processTimeMs, long lastAckTimestamp, int unfinished) Check if the packet process time shows that the relevant datanode is a slow node.static StreamSlowMonitor
void
onConfigurationChange
(org.apache.hadoop.conf.Configuration conf) This method would be called by theConfigurationManager
object when theConfiguration
object is reloaded from disk.private void
setConf
(org.apache.hadoop.conf.Configuration conf)
-
Field Details
-
LOG
-
WAL_SLOW_DETECT_MIN_COUNT_KEY
Configure for the min count for a datanode detected slow. If a datanode is detected slow times up to this count, then it will be added to the exclude datanode cache byExcludeDatanodeManager.tryAddExcludeDN(DatanodeInfo, String)
of this regionsever.- See Also:
-
DEFAULT_WAL_SLOW_DETECT_MIN_COUNT
- See Also:
-
WAL_SLOW_DETECT_DATA_TTL_KEY
Configure for the TTL of the data that a datanode detected slow.- See Also:
-
DEFAULT_WAL_SLOW_DETECT_DATA_TTL
- See Also:
-
DATANODE_PACKET_FLUSH_CHECK_SPEED_MIN_DATA_LENGTH_KEY
Configure for the speed check of packet min length. For packets whose data length smaller than this value, check slow by processing time. While for packets whose data length larger than this value, check slow by flushing speed.- See Also:
-
DEFAULT_DATANODE_PACKET_FLUSH_CHECK_SPEED_MIN_DATA_LENGTH
- See Also:
-
DATANODE_SLOW_PACKET_PROCESS_TIME_KEY
Configure for the slow packet process time, a duration from send to ACK. The processing time check is for packets that data length smaller thanDATANODE_PACKET_FLUSH_CHECK_SPEED_MIN_DATA_LENGTH_KEY
- See Also:
-
DEFAULT_DATANODE_SLOW_PACKET_PROCESS_TIME
- See Also:
-
DATANODE_SLOW_PACKET_FLUSH_MIN_SPEED_KEY
Configure for the check of large packet(which is configured byDATANODE_PACKET_FLUSH_CHECK_SPEED_MIN_DATA_LENGTH_KEY
) flush speed. e.g. If the configured slow packet process time is smaller than 10s, then here 20KB/s means 64KB should be processed in less than 3.2s.- See Also:
-
DEFAULT_DATANODE_SLOW_PACKET_FLUSH_MIN_SPEED
- See Also:
-
name
-
datanodeSlowDataQueue
private final org.apache.hbase.thirdparty.com.google.common.cache.LoadingCache<org.apache.hadoop.hdfs.protocol.DatanodeInfo,Deque<StreamSlowMonitor.PacketAckData>> datanodeSlowDataQueue -
excludeDatanodeManager
-
minSlowDetectCount
-
slowDataTtl
-
slowPacketAckMs
-
minPacketFlushSpeedKBs
-
minLengthForSpeedCheck
-
-
Constructor Details
-
StreamSlowMonitor
public StreamSlowMonitor(org.apache.hadoop.conf.Configuration conf, String name, ExcludeDatanodeManager excludeDatanodeManager)
-
-
Method Details
-
create
-
checkProcessTimeAndSpeed
public void checkProcessTimeAndSpeed(org.apache.hadoop.hdfs.protocol.DatanodeInfo datanodeInfo, long packetDataLen, long processTimeMs, long lastAckTimestamp, int unfinished) Check if the packet process time shows that the relevant datanode is a slow node.- Parameters:
datanodeInfo
- the datanode that processed the packetpacketDataLen
- the data length of the packet (in bytes)processTimeMs
- the process time (in ms) of the packet on the datanode,lastAckTimestamp
- the last acked timestamp of the packet on another datanodeunfinished
- if the packet is unfinished flushed to the datanode replicas
-
onConfigurationChange
Description copied from interface:ConfigurationObserver
This method would be called by theConfigurationManager
object when theConfiguration
object is reloaded from disk.- Specified by:
onConfigurationChange
in interfaceConfigurationObserver
-
addSlowAckData
private boolean addSlowAckData(org.apache.hadoop.hdfs.protocol.DatanodeInfo datanodeInfo, long dataLength, long processTime) -
setConf
-
getExcludeDatanodeManager
-