DefaultHeapMemoryTuner (Apache HBase 1.2.12 API)

java.lang.Object
- org.apache.hadoop.hbase.regionserver.DefaultHeapMemoryTuner

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, HeapMemoryTuner
```
@InterfaceAudience.Private
 class DefaultHeapMemoryTuner
extends Object
implements HeapMemoryTuner
```
The default implementation for the HeapMemoryTuner. This will do statistical checks on number of evictions, cache misses and flushes to decide whether there should be changes in the heap size of memstore/block cache. During each tuner operation tuner takes a step which can either be INCREASE_BLOCK_CACHE_SIZE (increase block cache size), INCREASE_MEMSTORE_SIZE (increase memstore size) and by default it is NEUTRAL (no change). We say block cache is sufficient when there is no block cache eviction at all or major amount of memory allocated to block cache is empty, similarly we say memory allocated for memstore is sufficient when there is no memstore flushes because of heap pressure or major amount of memory allocated to memstore is empty. If both are sufficient we do nothing, if exactly one of them is found to be sufficient we decrease its size by step and increase the other by same amount. If none of them is sufficient we do statistical analysis on number of cache misses and flushes to determine tuner direction. Based on these statistics we decide the tuner direction. If we are not confident about which step direction to take we do nothing and wait for next iteration. On expectation we will be tuning for at least 10% tuner calls. The number of past periods to consider for statistics calculation can be specified in config by hbase.regionserver.heapmemory.autotuner.lookup.periods. Also these many initial calls to tuner will be ignored (cache is warming up and we leave the system to reach steady state). After the tuner takes a step, in next call we insure that last call was indeed helpful and did not do us any harm. If not then we revert the previous step. The step size is dynamic and it changes based on current and past few tuning directions and their step sizes. We maintain a parameter decayingAvgTunerStepSize which is sum of past tuner steps with sign(positive for increase in memstore and negative for increase in block cache). But rather than simple sum it is calculated by giving more priority to the recent tuning steps. When last few tuner steps were NETURAL then we assume we are restarting the tuning process and step size is updated to maximum allowed size which can be specified in config by hbase.regionserver.heapmemory.autotuner.step.max. If in a particular tuning operation the step direction is opposite to what indicated by decayingTunerStepSizeSum we decrease the step size by half. Step size does not change in other tuning operations. When step size gets below a certain threshold then the following tuner operations are considered to be neutral. The minimum step size can be specified in config by hbase.regionserver.heapmemory.autotuner.step.min.

Nested Class Summary

Nested Classes
Modifier and Type Class and Description

private static class DefaultHeapMemoryTuner.StepDirection

Nested Classes
Modifier and Type	Class and Description
`private static class`	`DefaultHeapMemoryTuner.StepDirection`

Field Summary

Fields
Modifier and Type	Field and Description
`private float`	`blockCachePercentMaxRange`
`private float`	`blockCachePercentMinRange`
`private org.apache.hadoop.conf.Configuration`	`conf`
`private double`	`decayingTunerStepSizeSum`
`static int`	`DEFAULT_LOOKUP_PERIODS`
`static float`	`DEFAULT_MAX_STEP_VALUE`
`static float`	`DEFAULT_MIN_STEP_VALUE`
`static int`	`DEFAULT_NUM_PERIODS_IGNORED`
`static float`	`DEFAULT_SUFFICIENT_MEMORY_LEVEL_VALUE`
`private float`	`globalMemStorePercentMaxRange`
`private float`	`globalMemStorePercentMinRange`
`private int`	`ignoreInitialPeriods`
`private org.apache.commons.logging.Log`	`LOG`
`static String`	`LOOKUP_PERIODS_KEY`
`static String`	`MAX_STEP_KEY`
`private float`	`maximumStepSize`
`static String`	`MIN_STEP_KEY`
`private float`	`minimumStepSize`
`private static HeapMemoryManager.TunerResult`	`NO_OP_TUNER_RESULT`
`static String`	`NUM_PERIODS_TO_IGNORE`
`private int`	`numPeriodsToIgnore`
`private DefaultHeapMemoryTuner.StepDirection`	`prevTuneDirection`
`private RollingStatCalculator`	`rollingStatsForCacheMisses`
`private RollingStatCalculator`	`rollingStatsForEvictions`
`private RollingStatCalculator`	`rollingStatsForFlushes`
`private RollingStatCalculator`	`rollingStatsForTunerSteps`
`private float`	`step`
`static String`	`SUFFICIENT_MEMORY_LEVEL_KEY`
`private float`	`sufficientMemoryLevel`
`private HeapMemoryManager.TunerResult`	`TUNER_RESULT`
`private static double`	`TUNER_STEP_EPS`
`private int`	`tunerLookupPeriods`

Constructor Summary

Constructors
Constructor and Description

DefaultHeapMemoryTuner()

Constructors
Constructor and Description
`DefaultHeapMemoryTuner()`

Method Summary

Methods
Modifier and Type	Method and Description
`private void`	`addToRollingStats(HeapMemoryManager.TunerContext context)` Add the given context to the rolling tuner stats.
`org.apache.hadoop.conf.Configuration`	`getConf()`
`private DefaultHeapMemoryTuner.StepDirection`	`getTuneDirection(HeapMemoryManager.TunerContext context)` Determine best direction of tuning base on given context.
`void`	`setConf(org.apache.hadoop.conf.Configuration conf)`
`HeapMemoryManager.TunerResult`	`tune(HeapMemoryManager.TunerContext context)` Perform the heap memory tuning operation.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MAX_STEP_KEY

public static final String MAX_STEP_KEY

See Also:: Constant Field Values

MIN_STEP_KEY

public static final String MIN_STEP_KEY

See Also:: Constant Field Values

SUFFICIENT_MEMORY_LEVEL_KEY

public static final String SUFFICIENT_MEMORY_LEVEL_KEY

See Also:: Constant Field Values

LOOKUP_PERIODS_KEY

public static final String LOOKUP_PERIODS_KEY

See Also:: Constant Field Values

NUM_PERIODS_TO_IGNORE

public static final String NUM_PERIODS_TO_IGNORE

See Also:: Constant Field Values

DEFAULT_MAX_STEP_VALUE
```
public static final float DEFAULT_MAX_STEP_VALUE
```
See Also:
Constant Field Values

DEFAULT_MIN_STEP_VALUE
```
public static final float DEFAULT_MIN_STEP_VALUE
```
See Also:
Constant Field Values

DEFAULT_SUFFICIENT_MEMORY_LEVEL_VALUE
```
public static final float DEFAULT_SUFFICIENT_MEMORY_LEVEL_VALUE
```
See Also:
Constant Field Values

DEFAULT_LOOKUP_PERIODS
```
public static final int DEFAULT_LOOKUP_PERIODS
```
See Also:
Constant Field Values

DEFAULT_NUM_PERIODS_IGNORED
```
public static final int DEFAULT_NUM_PERIODS_IGNORED
```
See Also:
Constant Field Values

NO_OP_TUNER_RESULT

private static final HeapMemoryManager.TunerResult NO_OP_TUNER_RESULT

TUNER_STEP_EPS
```
private static final double TUNER_STEP_EPS
```
See Also:
Constant Field Values

LOG

private org.apache.commons.logging.Log LOG

TUNER_RESULT

private HeapMemoryManager.TunerResult TUNER_RESULT

conf

private org.apache.hadoop.conf.Configuration conf

sufficientMemoryLevel
```
private float sufficientMemoryLevel
```

maximumStepSize
```
private float maximumStepSize
```

minimumStepSize
```
private float minimumStepSize
```

tunerLookupPeriods
```
private int tunerLookupPeriods
```

numPeriodsToIgnore
```
private int numPeriodsToIgnore
```

ignoreInitialPeriods
```
private int ignoreInitialPeriods
```

globalMemStorePercentMinRange

private float globalMemStorePercentMinRange

globalMemStorePercentMaxRange

private float globalMemStorePercentMaxRange

blockCachePercentMinRange

private float blockCachePercentMinRange

blockCachePercentMaxRange

private float blockCachePercentMaxRange

rollingStatsForCacheMisses

private RollingStatCalculator rollingStatsForCacheMisses

rollingStatsForFlushes

private RollingStatCalculator rollingStatsForFlushes

rollingStatsForEvictions

private RollingStatCalculator rollingStatsForEvictions

rollingStatsForTunerSteps

private RollingStatCalculator rollingStatsForTunerSteps

step
```
private float step
```

prevTuneDirection

private DefaultHeapMemoryTuner.StepDirection prevTuneDirection

decayingTunerStepSizeSum

private double decayingTunerStepSizeSum

Constructor Detail
- DefaultHeapMemoryTuner
```
DefaultHeapMemoryTuner()
```

Method Detail
- tune
```
public HeapMemoryManager.TunerResult tune(HeapMemoryManager.TunerContext context)
```
  Description copied from interface: HeapMemoryTuner
  
  Perform the heap memory tuning operation.
  
  Specified by:
  
  tune in interface HeapMemoryTuner
  
  Returns:
  TunerResult including the heap percentage for memstore and block cache
- getTuneDirection
```
private DefaultHeapMemoryTuner.StepDirection getTuneDirection(HeapMemoryManager.TunerContext context)
```
  Determine best direction of tuning base on given context.
  
  Parameters:
  context - The tuner context.
  
  Returns:
  tuning direction.
- addToRollingStats
```
private void addToRollingStats(HeapMemoryManager.TunerContext context)
```
  Add the given context to the rolling tuner stats.
  
  Parameters:
  context - The tuner context.
- getConf
```
public org.apache.hadoop.conf.Configuration getConf()
```
  Specified by:
  
  getConf in interface org.apache.hadoop.conf.Configurable
- setConf
```
public void setConf(org.apache.hadoop.conf.Configuration conf)
```
  Specified by:
  
  setConf in interface org.apache.hadoop.conf.Configurable

Class DefaultHeapMemoryTuner

Nested Class Summary

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

MAX_STEP_KEY

MIN_STEP_KEY

SUFFICIENT_MEMORY_LEVEL_KEY

LOOKUP_PERIODS_KEY

NUM_PERIODS_TO_IGNORE

DEFAULT_MAX_STEP_VALUE

DEFAULT_MIN_STEP_VALUE

DEFAULT_SUFFICIENT_MEMORY_LEVEL_VALUE

DEFAULT_LOOKUP_PERIODS

DEFAULT_NUM_PERIODS_IGNORED

NO_OP_TUNER_RESULT

TUNER_STEP_EPS

LOG

TUNER_RESULT

conf

sufficientMemoryLevel

maximumStepSize

minimumStepSize

tunerLookupPeriods

numPeriodsToIgnore

ignoreInitialPeriods

globalMemStorePercentMinRange

globalMemStorePercentMaxRange

blockCachePercentMinRange

blockCachePercentMaxRange

rollingStatsForCacheMisses

rollingStatsForFlushes

rollingStatsForEvictions

rollingStatsForTunerSteps

step

prevTuneDirection

decayingTunerStepSizeSum

Constructor Detail

DefaultHeapMemoryTuner

Method Detail

tune

getTuneDirection

addToRollingStats

getConf

setConf