org.apache.hadoop.hbase.regionserver

Class DefaultHeapMemoryTuner

java.lang.Object

org.apache.hadoop.hbase.regionserver.DefaultHeapMemoryTuner

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, HeapMemoryTuner

@InterfaceAudience.Private
class DefaultHeapMemoryTuner
extends Object
implements HeapMemoryTuner

The default implementation for the HeapMemoryTuner. This will do statistical checks on number of evictions, cache misses and flushes to decide whether there should be changes in the heap size of memstore/block cache. During each tuner operation tuner takes a step which can either be INCREASE_BLOCK_CACHE_SIZE (increase block cache size), INCREASE_MEMSTORE_SIZE (increase memstore size) and by default it is NEUTRAL (no change). We say block cache is sufficient when there is no block cache eviction at all or major amount of memory allocated to block cache is empty, similarly we say memory allocated for memstore is sufficient when there is no memstore flushes because of heap pressure or major amount of memory allocated to memstore is empty. If both are sufficient we do nothing, if exactly one of them is found to be sufficient we decrease its size by step and increase the other by same amount. If none of them is sufficient we do statistical analysis on number of cache misses and flushes to determine tuner direction. Based on these statistics we decide the tuner direction. If we are not confident about which step direction to take we do nothing and wait for next iteration. On expectation we will be tuning for at least 10% tuner calls. The number of past periods to consider for statistics calculation can be specified in config by hbase.regionserver.heapmemory.autotuner.lookup.periods. Also these many initial calls to tuner will be ignored (cache is warming up and we leave the system to reach steady state). After the tuner takes a step, in next call we insure that last call was indeed helpful and did not do us any harm. If not then we revert the previous step. The step size is dynamic and it changes based on current and past few tuning directions and their step sizes. We maintain a parameter decayingAvgTunerStepSize which is sum of past tuner steps with sign(positive for increase in memstore and negative for increase in block cache). But rather than simple sum it is calculated by giving more priority to the recent tuning steps. When last few tuner steps were NETURAL then we assume we are restarting the tuning process and step size is updated to maximum allowed size which can be specified in config by hbase.regionserver.heapmemory.autotuner.step.max. If in a particular tuning operation the step direction is opposite to what indicated by decayingTunerStepSizeSum we decrease the step size by half. Step size does not change in other tuning operations. When step size gets below a certain threshold then the following tuner operations are considered to be neutral. The minimum step size can be specified in config by hbase.regionserver.heapmemory.autotuner.step.min.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
private static class	DefaultHeapMemoryTuner.StepDirection

Field Summary

Fields

Modifier and Type	Field and Description
private float	blockCachePercentMaxRange
private float	blockCachePercentMinRange
private org.apache.hadoop.conf.Configuration	conf
private double	decayingTunerStepSizeSum
static int	DEFAULT_LOOKUP_PERIODS
static float	DEFAULT_MAX_STEP_VALUE
static float	DEFAULT_MIN_STEP_VALUE
static int	DEFAULT_NUM_PERIODS_IGNORED
static float	DEFAULT_SUFFICIENT_MEMORY_LEVEL_VALUE
private float	globalMemStoreLimitLowMarkPercent
private float	globalMemStorePercentMaxRange
private float	globalMemStorePercentMinRange
private int	ignoreInitialPeriods
private org.slf4j.Logger	LOG
static String	LOOKUP_PERIODS_KEY
static String	MAX_STEP_KEY
private float	maximumStepSize
static String	MIN_STEP_KEY
private float	minimumStepSize
private static HeapMemoryManager.TunerResult	NO_OP_TUNER_RESULT
static String	NUM_PERIODS_TO_IGNORE
private int	numPeriodsToIgnore
private DefaultHeapMemoryTuner.StepDirection	prevTuneDirection
private RollingStatCalculator	rollingStatsForCacheMisses
private RollingStatCalculator	rollingStatsForEvictions
private RollingStatCalculator	rollingStatsForFlushes
private RollingStatCalculator	rollingStatsForTunerSteps
private float	step
static String	SUFFICIENT_MEMORY_LEVEL_KEY
private float	sufficientMemoryLevel
private HeapMemoryManager.TunerResult	TUNER_RESULT
private static double	TUNER_STEP_EPS
private int	tunerLookupPeriods

Constructor Summary

Constructors

Constructor and Description
DefaultHeapMemoryTuner()

Method Summary

Modifier and Type	Method and Description
private void	addToRollingStats(HeapMemoryManager.TunerContext context) Add the given context to the rolling tuner stats.
org.apache.hadoop.conf.Configuration	getConf()
private DefaultHeapMemoryTuner.StepDirection	getTuneDirection(HeapMemoryManager.TunerContext context) Determine best direction of tuning base on given context.
void	setConf(org.apache.hadoop.conf.Configuration conf)
HeapMemoryManager.TunerResult	tune(HeapMemoryManager.TunerContext context) Perform the heap memory tuning operation.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MAX_STEP_KEY

public static final String MAX_STEP_KEY

See Also:

Constant Field Values

MIN_STEP_KEY

public static final String MIN_STEP_KEY

See Also:

Constant Field Values

SUFFICIENT_MEMORY_LEVEL_KEY

public static final String SUFFICIENT_MEMORY_LEVEL_KEY

See Also:

Constant Field Values

LOOKUP_PERIODS_KEY

public static final String LOOKUP_PERIODS_KEY

See Also:

Constant Field Values

NUM_PERIODS_TO_IGNORE

public static final String NUM_PERIODS_TO_IGNORE

See Also:

Constant Field Values

DEFAULT_MAX_STEP_VALUE

public static final float DEFAULT_MAX_STEP_VALUE

See Also:

Constant Field Values

DEFAULT_MIN_STEP_VALUE

public static final float DEFAULT_MIN_STEP_VALUE

See Also:

Constant Field Values

DEFAULT_SUFFICIENT_MEMORY_LEVEL_VALUE

public static final float DEFAULT_SUFFICIENT_MEMORY_LEVEL_VALUE

See Also:

Constant Field Values

DEFAULT_LOOKUP_PERIODS

public static final int DEFAULT_LOOKUP_PERIODS

See Also:

Constant Field Values

DEFAULT_NUM_PERIODS_IGNORED

public static final int DEFAULT_NUM_PERIODS_IGNORED

See Also:

Constant Field Values

NO_OP_TUNER_RESULT

private static final HeapMemoryManager.TunerResult NO_OP_TUNER_RESULT

TUNER_STEP_EPS

private static final double TUNER_STEP_EPS

See Also:

Constant Field Values

LOG

private org.slf4j.Logger LOG

TUNER_RESULT

private HeapMemoryManager.TunerResult TUNER_RESULT

conf

private org.apache.hadoop.conf.Configuration conf

sufficientMemoryLevel

private float sufficientMemoryLevel

maximumStepSize

private float maximumStepSize

minimumStepSize

private float minimumStepSize

tunerLookupPeriods

private int tunerLookupPeriods

numPeriodsToIgnore

private int numPeriodsToIgnore

ignoreInitialPeriods

private int ignoreInitialPeriods

globalMemStorePercentMinRange

private float globalMemStorePercentMinRange

globalMemStorePercentMaxRange

private float globalMemStorePercentMaxRange

blockCachePercentMinRange

private float blockCachePercentMinRange

blockCachePercentMaxRange

private float blockCachePercentMaxRange

globalMemStoreLimitLowMarkPercent

private float globalMemStoreLimitLowMarkPercent

rollingStatsForCacheMisses

private RollingStatCalculator rollingStatsForCacheMisses

rollingStatsForFlushes

private RollingStatCalculator rollingStatsForFlushes

rollingStatsForEvictions

private RollingStatCalculator rollingStatsForEvictions

rollingStatsForTunerSteps

private RollingStatCalculator rollingStatsForTunerSteps

step

private float step

prevTuneDirection

private DefaultHeapMemoryTuner.StepDirection prevTuneDirection

decayingTunerStepSizeSum

private double decayingTunerStepSizeSum

Constructor Detail

DefaultHeapMemoryTuner

DefaultHeapMemoryTuner()

Method Detail

tune

public HeapMemoryManager.TunerResult tune(HeapMemoryManager.TunerContext context)

Description copied from interface: HeapMemoryTuner

Perform the heap memory tuning operation.

Specified by:

tune in interface HeapMemoryTuner

Returns:

TunerResult including the heap percentage for memstore and block cache

getTuneDirection

private DefaultHeapMemoryTuner.StepDirection getTuneDirection(HeapMemoryManager.TunerContext context)

Determine best direction of tuning base on given context.

Parameters:

context - The tuner context.

Returns:

tuning direction.

addToRollingStats

private void addToRollingStats(HeapMemoryManager.TunerContext context)

Add the given context to the rolling tuner stats.

Parameters:

context - The tuner context.

getConf

public org.apache.hadoop.conf.Configuration getConf()

Specified by:

getConf in interface org.apache.hadoop.conf.Configurable

setConf

public void setConf(org.apache.hadoop.conf.Configuration conf)

Specified by:

setConf in interface org.apache.hadoop.conf.Configurable