org.apache.hadoop.hbase.regionserver.throttle

Class StoreHotnessProtector

java.lang.Object

org.apache.hadoop.hbase.regionserver.throttle.StoreHotnessProtector

@InterfaceAudience.Private
public class StoreHotnessProtector
extends Object

StoreHotnessProtector is designed to help limit the concurrency of puts with dense columns, it does best-effort to avoid exhausting all RS's handlers. When a lot of clients write requests with dense (hundreds) columns to a Store at the same time, it will lead to blocking of RS because CSLM degrades when concurrency goes up. It's not a kind of throttling. Throttling is user-oriented, while StoreHotnessProtector is system-oriented, RS-self-protected mechanism.

There are three key parameters:

1. parallelPutToStoreThreadLimitCheckMinColumnCount: If the amount of columns exceed this threshold, the HotProtector will work, 100 by default

2. parallelPutToStoreThreadLimit: The amount of concurrency allowed to write puts to a Store at the same time.

3. parallelPreparePutToStoreThreadLimit: The amount of concurrency allowed to prepare writing puts to a Store at the same time.

Notice that our writing pipeline includes three key process: MVCC acquire, writing MemStore, and WAL. Only limit the concurrency of writing puts to Store(parallelPutToStoreThreadLimit) is not enough since the actual concurrency of puts may still exceed the limit when MVCC contention or slow WAL sync happens. This is why parallelPreparePutToStoreThreadLimit is needed.

This protector is enabled by default and could be turned off by setting hbase.region.store.parallel.put.limit to 0, supporting online configuration change.

Field Summary

Fields

Modifier and Type	Field and Description
private static int	DEFAULT_PARALLEL_PREPARE_PUT_STORE_MULTIPLIER
private static int	DEFAULT_PARALLEL_PUT_STORE_THREADS_LIMIT
private static int	DEFAULT_PARALLEL_PUT_STORE_THREADS_LIMIT_MIN_COLUMN_NUM
static long	FIXED_SIZE
private static org.slf4j.Logger	LOG
private static boolean	loggedDisableMessage
static String	PARALLEL_PREPARE_PUT_STORE_MULTIPLIER
static String	PARALLEL_PUT_STORE_THREADS_LIMIT
static String	PARALLEL_PUT_STORE_THREADS_LIMIT_MIN_COLUMN_COUNT
private int	parallelPreparePutToStoreThreadLimit
private int	parallelPutToStoreThreadLimit
private int	parallelPutToStoreThreadLimitCheckMinColumnCount
private Map<byte[],AtomicInteger>	preparePutToStoreMap
private Region	region

Constructor Summary

Constructors

Constructor and Description
StoreHotnessProtector(Region region, org.apache.hadoop.conf.Configuration conf)

Method Summary

Modifier and Type	Method and Description
void	finish(Map<byte[],List<Cell>> familyMaps)
(package private) Map<byte[],AtomicInteger>	getPreparePutToStoreMap()
void	init(org.apache.hadoop.conf.Configuration conf)
boolean	isEnable()
private static void	logDisabledMessageOnce() init(Configuration) is called for every Store that opens on a RegionServer.
void	start(Map<byte[],List<Cell>> familyMaps)
String	toString()
void	update(org.apache.hadoop.conf.Configuration conf)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

LOG

private static final org.slf4j.Logger LOG

loggedDisableMessage

private static volatile boolean loggedDisableMessage

parallelPutToStoreThreadLimit

private volatile int parallelPutToStoreThreadLimit

parallelPreparePutToStoreThreadLimit

private volatile int parallelPreparePutToStoreThreadLimit

PARALLEL_PUT_STORE_THREADS_LIMIT

public static final String PARALLEL_PUT_STORE_THREADS_LIMIT

See Also:

Constant Field Values

PARALLEL_PREPARE_PUT_STORE_MULTIPLIER

public static final String PARALLEL_PREPARE_PUT_STORE_MULTIPLIER

See Also:

Constant Field Values

DEFAULT_PARALLEL_PUT_STORE_THREADS_LIMIT

private static final int DEFAULT_PARALLEL_PUT_STORE_THREADS_LIMIT

See Also:

Constant Field Values

parallelPutToStoreThreadLimitCheckMinColumnCount

private volatile int parallelPutToStoreThreadLimitCheckMinColumnCount

PARALLEL_PUT_STORE_THREADS_LIMIT_MIN_COLUMN_COUNT

public static final String PARALLEL_PUT_STORE_THREADS_LIMIT_MIN_COLUMN_COUNT

See Also:

Constant Field Values

DEFAULT_PARALLEL_PUT_STORE_THREADS_LIMIT_MIN_COLUMN_NUM

private static final int DEFAULT_PARALLEL_PUT_STORE_THREADS_LIMIT_MIN_COLUMN_NUM

See Also:

Constant Field Values

DEFAULT_PARALLEL_PREPARE_PUT_STORE_MULTIPLIER

private static final int DEFAULT_PARALLEL_PREPARE_PUT_STORE_MULTIPLIER

See Also:

Constant Field Values

preparePutToStoreMap

private final Map<byte[],AtomicInteger> preparePutToStoreMap

region

private final Region region

FIXED_SIZE

public static final long FIXED_SIZE

Constructor Detail

StoreHotnessProtector

public StoreHotnessProtector(Region region,
                             org.apache.hadoop.conf.Configuration conf)

Method Detail

init

public void init(org.apache.hadoop.conf.Configuration conf)

logDisabledMessageOnce

private static void logDisabledMessageOnce()

init(Configuration) is called for every Store that opens on a RegionServer. Here we make a lightweight attempt to log this message once per RegionServer, rather than per-Store. The goal is just to draw attention to this feature if debugging overload due to heavy writes.

update

public void update(org.apache.hadoop.conf.Configuration conf)

start

public void start(Map<byte[],List<Cell>> familyMaps)
           throws RegionTooBusyException

Throws:

RegionTooBusyException

finish

public void finish(Map<byte[],List<Cell>> familyMaps)

toString

public String toString()

Overrides:

toString in class Object

isEnable

public boolean isEnable()

getPreparePutToStoreMap

Map<byte[],AtomicInteger> getPreparePutToStoreMap()