StochasticLoadBalancer (Apache HBase 2.2.3 API)

java.lang.Object
- org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer
- - org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, ConfigurationObserver, LoadBalancer, Stoppable

Direct Known Subclasses:

FavoredStochasticBalancer
```
@InterfaceAudience.LimitedPrivate(value="Configuration")
public class StochasticLoadBalancer
extends BaseLoadBalancer
```
This is a best effort load balancer. Given a Cost function F(C) => x It will randomly try and mutate the cluster to Cprime. If F(Cprime) < F(C) then the new cluster state becomes the plan. It includes costs functions to compute the cost of:
- Region Load
- Table Load
- Data Locality
- Memstore Sizes
- Storefile Sizes
Every cost function returns a number between 0 and 1 inclusive; where 0 is the lowest cost best solution, and 1 is the highest possible cost and the worst solution. The computed costs are scaled by their respective multipliers:
- hbase.master.balancer.stochastic.regionLoadCost
- hbase.master.balancer.stochastic.moveCost
- hbase.master.balancer.stochastic.tableLoadCost
- hbase.master.balancer.stochastic.localityCost
- hbase.master.balancer.stochastic.memstoreSizeCost
- hbase.master.balancer.stochastic.storefileSizeCost
In addition to the above configurations, the balancer can be tuned by the following configuration values:
- hbase.master.balancer.stochastic.maxMoveRegions which controls what the max number of regions that can be moved in a single invocation of this balancer.
- hbase.master.balancer.stochastic.stepsPerRegion is the coefficient by which the number of regions is multiplied to try and get the number of times the balancer will mutate all servers.
- hbase.master.balancer.stochastic.maxSteps which controls the maximum number of times that the balancer will try and mutate all the servers. The balancer will use the minimum of this value and the above computation.
This balancer is best used with hbase.master.loadbalance.bytable set to false so that the balancer gets the full picture of all loads on the cluster.

Nested Class Summary

Nested Classes
Modifier and Type	Class and Description
`(package private) static class`	`StochasticLoadBalancer.CandidateGenerator` Generates a candidate action to be applied to the cluster for cost function search
`(package private) static class`	`StochasticLoadBalancer.CostFromRegionLoadAsRateFunction` Class to be used for the subset of RegionLoad costs that should be treated as rates.
`(package private) static class`	`StochasticLoadBalancer.CostFromRegionLoadFunction` Base class the allows writing costs functions from rolling average of some number from RegionLoad.
`(package private) static class`	`StochasticLoadBalancer.CostFunction` Base class of StochasticLoadBalancer's Cost Functions.
`(package private) static class`	`StochasticLoadBalancer.LoadCandidateGenerator`
`(package private) static class`	`StochasticLoadBalancer.LocalityBasedCandidateGenerator`
`(package private) static class`	`StochasticLoadBalancer.LocalityBasedCostFunction` Compute a cost of a potential cluster configuration based upon where `HStoreFile`s are located.
`(package private) static class`	`StochasticLoadBalancer.MemStoreSizeCostFunction` Compute the cost of total memstore size.
`(package private) static class`	`StochasticLoadBalancer.MoveCostFunction` Given the starting state of the regions and a potential ending state compute cost based upon the number of regions that have moved.
`(package private) static class`	`StochasticLoadBalancer.PrimaryRegionCountSkewCostFunction` Compute the cost of a potential cluster state from skew in number of primary regions on a cluster.
`(package private) static class`	`StochasticLoadBalancer.RackLocalityCostFunction`
`(package private) static class`	`StochasticLoadBalancer.RandomCandidateGenerator`
`(package private) static class`	`StochasticLoadBalancer.ReadRequestCostFunction` Compute the cost of total number of read requests The more unbalanced the higher the computed cost will be.
`(package private) static class`	`StochasticLoadBalancer.RegionCountSkewCostFunction` Compute the cost of a potential cluster state from skew in number of regions on a cluster.
`(package private) static class`	`StochasticLoadBalancer.RegionReplicaCandidateGenerator` Generates candidates which moves the replicas out of the region server for co-hosted region replicas
`(package private) static class`	`StochasticLoadBalancer.RegionReplicaHostCostFunction` A cost function for region replicas.
`(package private) static class`	`StochasticLoadBalancer.RegionReplicaRackCandidateGenerator` Generates candidates which moves the replicas out of the rack for co-hosted region replicas in the same rack
`(package private) static class`	`StochasticLoadBalancer.RegionReplicaRackCostFunction` A cost function for region replicas for the rack distribution.
`(package private) static class`	`StochasticLoadBalancer.ServerLocalityCostFunction`
`(package private) static class`	`StochasticLoadBalancer.StoreFileCostFunction` Compute the cost of total open storefiles size.
`(package private) static class`	`StochasticLoadBalancer.TableSkewCostFunction` Compute the cost of a potential cluster configuration based upon how evenly distributed tables are.
`(package private) static class`	`StochasticLoadBalancer.WriteRequestCostFunction` Compute the cost of total number of write requests.

Nested classes/interfaces inherited from class org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer
BaseLoadBalancer.Cluster

Field Summary

Fields
Modifier and Type	Field and Description
`private List<StochasticLoadBalancer.CandidateGenerator>`	`candidateGenerators`
`private StochasticLoadBalancer.CostFunction[]`	`costFunctions`
`private Double[]`	`curFunctionCosts`
`private Double`	`curOverallCost`
`private boolean`	`isByTable`
`protected static String`	`KEEP_REGION_LOADS`
`(package private) Map<String,Deque<BalancerRegionLoad>>`	`loads`
`private StochasticLoadBalancer.LocalityBasedCandidateGenerator`	`localityCandidateGenerator`
`private StochasticLoadBalancer.ServerLocalityCostFunction`	`localityCost`
`private static org.slf4j.Logger`	`LOG`
`protected static String`	`MAX_RUNNING_TIME_KEY`
`protected static String`	`MAX_STEPS_KEY`
`private long`	`maxRunningTime`
`private int`	`maxSteps`
`protected static String`	`MIN_COST_NEED_BALANCE_KEY`
`private float`	`minCostNeedBalance`
`private int`	`numRegionLoadsToRemember`
`private StochasticLoadBalancer.RackLocalityCostFunction`	`rackLocalityCost`
`protected static Random`	`RANDOM`
`private StochasticLoadBalancer.CostFromRegionLoadFunction[]`	`regionLoadFunctions`
`private StochasticLoadBalancer.RegionReplicaHostCostFunction`	`regionReplicaHostCostFunction`
`private StochasticLoadBalancer.RegionReplicaRackCostFunction`	`regionReplicaRackCostFunction`
`protected static String`	`RUN_MAX_STEPS_KEY`
`private boolean`	`runMaxSteps`
`protected static String`	`STEPS_PER_REGION_KEY`
`private int`	`stepsPerRegion`
`private static String`	`TABLE_FUNCTION_SEP`
`private TableName`	`tableName`
`private Double[]`	`tempFunctionCosts`

Fields inherited from class org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer
clusterStatus, config, IDLE_SERVER_PREDICATOR, maintenanceMode, masterServerName, metricsBalancer, MIN_SERVER_BALANCE, onlySystemTablesOnMaster, overallSlop, rackManager, regionFinder, services, slop, useRegionFinder

Fields inherited from interface org.apache.hadoop.hbase.master.LoadBalancer
BOGUS_SERVER_NAME, SYSTEM_TABLES_ON_MASTER, TABLES_ON_MASTER

Constructor Summary

Constructors
Constructor and Description

StochasticLoadBalancer()
The constructor that pass a MetricsStochasticBalancer to BaseLoadBalancer to replace its default MetricsBalancer

Constructors
Constructor and Description
`StochasticLoadBalancer()` The constructor that pass a MetricsStochasticBalancer to BaseLoadBalancer to replace its default MetricsBalancer

Method Summary

All Methods Static Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`protected boolean`	`areSomeRegionReplicasColocated(BaseLoadBalancer.Cluster c)` Subclasses should implement this to return true if the cluster has nodes that hosts multiple replicas for the same region, or, if there are multiple racks and the same rack hosts replicas of the same region
`List<RegionPlan>`	`balanceCluster(Map<ServerName,List<RegionInfo>> clusterState)` Given the cluster state this will try and approach an optimal balance.
`List<RegionPlan>`	`balanceCluster(TableName tableName, Map<ServerName,List<RegionInfo>> clusterState)` Perform the major balance operation
`static String`	`composeAttributeName(String tableName, String costFunctionName)` A helper function to compose the attribute name from tablename and costfunction name
`protected double`	`computeCost(BaseLoadBalancer.Cluster cluster, double previousCost)` This is the main cost function.
`private List<RegionPlan>`	`createRegionPlans(BaseLoadBalancer.Cluster cluster)` Create all of the RegionPlan's needed to move from the initial cluster state to the desired state.
`private String`	`functionCost()`
`String[]`	`getCostFunctionNames()` Get the names of the cost functions
`protected void`	`initCosts(BaseLoadBalancer.Cluster cluster)`
`protected boolean`	`needsBalance(BaseLoadBalancer.Cluster cluster)`
`(package private) BaseLoadBalancer.Cluster.Action`	`nextAction(BaseLoadBalancer.Cluster cluster)`
`void`	`onConfigurationChange(org.apache.hadoop.conf.Configuration conf)` This method would be called by the `ConfigurationManager` object when the `Configuration` object is reloaded from disk.
`protected void`	`setCandidateGenerators(List<StochasticLoadBalancer.CandidateGenerator> customCandidateGenerators)`
`void`	`setClusterMetrics(ClusterMetrics st)` Set the current cluster status.
`void`	`setConf(org.apache.hadoop.conf.Configuration conf)`
`void`	`setMasterServices(MasterServices masterServices)` Set the master service.
`protected void`	`setSlop(org.apache.hadoop.conf.Configuration conf)`
`protected void`	`updateCostsWithAction(BaseLoadBalancer.Cluster cluster, BaseLoadBalancer.Cluster.Action action)`
`void`	`updateMetricsSize(int size)` Update the number of metrics that are reported to JMX
`private void`	`updateRegionLoad()` Store the current region loads.
`private void`	`updateStochasticCosts(TableName tableName, Double overall, Double[] subCosts)` update costs to JMX

Methods inherited from class org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer
assignMasterSystemRegions, balanceMasterRegions, createCluster, getConf, getRegionAssignmentsByServer, initialize, isStopped, postMasterStartupInitialize, randomAssignment, regionOffline, regionOnline, retainAssignment, roundRobinAssignment, setClusterLoad, setRackManager, shouldBeOnMaster, stop, updateBalancerStatus

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.hbase.master.LoadBalancer
isMasterCanHostUserRegions, isSystemTablesOnlyOnMaster, isTablesOnMaster

- Field Detail
  - STEPS_PER_REGION_KEY
```
protected static final String STEPS_PER_REGION_KEY
```
    See Also:
    
    Constant Field Values
  - MAX_STEPS_KEY
```
protected static final String MAX_STEPS_KEY
```
    See Also:
    
    Constant Field Values
  - RUN_MAX_STEPS_KEY
```
protected static final String RUN_MAX_STEPS_KEY
```
    See Also:
    
    Constant Field Values
  - MAX_RUNNING_TIME_KEY
```
protected static final String MAX_RUNNING_TIME_KEY
```
    See Also:
    
    Constant Field Values
  - KEEP_REGION_LOADS
```
protected static final String KEEP_REGION_LOADS
```
    See Also:
    
    Constant Field Values
  - TABLE_FUNCTION_SEP
```
private static final String TABLE_FUNCTION_SEP
```
    See Also:
    
    Constant Field Values
  - MIN_COST_NEED_BALANCE_KEY
```
protected static final String MIN_COST_NEED_BALANCE_KEY
```
    See Also:
    
    Constant Field Values
  - RANDOM
```
protected static final Random RANDOM
```
  - LOG
```
private static final org.slf4j.Logger LOG
```
  - loads
```
Map<String,Deque<BalancerRegionLoad>> loads
```
  - maxSteps
```
private int maxSteps
```
  - runMaxSteps
```
private boolean runMaxSteps
```
  - stepsPerRegion
```
private int stepsPerRegion
```
  - maxRunningTime
```
private long maxRunningTime
```
  - numRegionLoadsToRemember
```
private int numRegionLoadsToRemember
```
  - minCostNeedBalance
```
private float minCostNeedBalance
```
  - candidateGenerators
```
private List<StochasticLoadBalancer.CandidateGenerator> candidateGenerators
```
  - regionLoadFunctions
```
private StochasticLoadBalancer.CostFromRegionLoadFunction[] regionLoadFunctions
```
  - costFunctions
```
private StochasticLoadBalancer.CostFunction[] costFunctions
```
  - curOverallCost
```
private Double curOverallCost
```
  - tempFunctionCosts
```
private Double[] tempFunctionCosts
```
  - curFunctionCosts
```
private Double[] curFunctionCosts
```
  - localityCandidateGenerator
```
private StochasticLoadBalancer.LocalityBasedCandidateGenerator localityCandidateGenerator
```
  - localityCost
```
private StochasticLoadBalancer.ServerLocalityCostFunction localityCost
```
  - rackLocalityCost
```
private StochasticLoadBalancer.RackLocalityCostFunction rackLocalityCost
```
  - regionReplicaHostCostFunction
```
private StochasticLoadBalancer.RegionReplicaHostCostFunction regionReplicaHostCostFunction
```
  - regionReplicaRackCostFunction
```
private StochasticLoadBalancer.RegionReplicaRackCostFunction regionReplicaRackCostFunction
```
  - isByTable
```
private boolean isByTable
```
  - tableName
```
private TableName tableName
```
- Constructor Detail
  - StochasticLoadBalancer
```
public StochasticLoadBalancer()
```
    The constructor that pass a MetricsStochasticBalancer to BaseLoadBalancer to replace its default MetricsBalancer
- Method Detail
  - onConfigurationChange
```
public void onConfigurationChange(org.apache.hadoop.conf.Configuration conf)
```
    Description copied from interface: ConfigurationObserver
    
    This method would be called by the ConfigurationManager object when the Configuration object is reloaded from disk.
    
    Specified by:
    
    onConfigurationChange in interface ConfigurationObserver
    
    Specified by:
    
    onConfigurationChange in interface LoadBalancer
    
    Overrides:
    
    onConfigurationChange in class BaseLoadBalancer
  - setConf
```
public void setConf(org.apache.hadoop.conf.Configuration conf)
```
    Specified by:
    
    setConf in interface org.apache.hadoop.conf.Configurable
    
    Overrides:
    
    setConf in class BaseLoadBalancer
  - setCandidateGenerators
```
protected void setCandidateGenerators(List<StochasticLoadBalancer.CandidateGenerator> customCandidateGenerators)
```
  - setSlop
```
protected void setSlop(org.apache.hadoop.conf.Configuration conf)
```
    Overrides:
    
    setSlop in class BaseLoadBalancer
  - setClusterMetrics
```
public void setClusterMetrics(ClusterMetrics st)
```
    Description copied from interface: LoadBalancer
    
    Set the current cluster status. This allows a LoadBalancer to map host name to a server
    
    Specified by:
    
    setClusterMetrics in interface LoadBalancer
    
    Overrides:
    
    setClusterMetrics in class BaseLoadBalancer
  - updateMetricsSize
```
public void updateMetricsSize(int size)
```
    Update the number of metrics that are reported to JMX
  - setMasterServices
```
public void setMasterServices(MasterServices masterServices)
```
    Description copied from interface: LoadBalancer
    
    Set the master service.
    
    Specified by:
    
    setMasterServices in interface LoadBalancer
    
    Overrides:
    
    setMasterServices in class BaseLoadBalancer
  - areSomeRegionReplicasColocated
```
protected boolean areSomeRegionReplicasColocated(BaseLoadBalancer.Cluster c)
```
    Description copied from class: BaseLoadBalancer
    
    Subclasses should implement this to return true if the cluster has nodes that hosts multiple replicas for the same region, or, if there are multiple racks and the same rack hosts replicas of the same region
    
    Overrides:
    
    areSomeRegionReplicasColocated in class BaseLoadBalancer
    
    Parameters:
    
    c - Cluster information
    
    Returns:
    
    whether region replicas are currently co-located
  - needsBalance
```
protected boolean needsBalance(BaseLoadBalancer.Cluster cluster)
```
    Overrides:
    
    needsBalance in class BaseLoadBalancer
  - balanceCluster
```
public List<RegionPlan> balanceCluster(TableName tableName,
                                       Map<ServerName,List<RegionInfo>> clusterState)
```
    Description copied from interface: LoadBalancer
    
    Perform the major balance operation
    
    Returns:
    
    List of plans
  - nextAction
```
BaseLoadBalancer.Cluster.Action nextAction(BaseLoadBalancer.Cluster cluster)
```
  - balanceCluster
```
public List<RegionPlan> balanceCluster(Map<ServerName,List<RegionInfo>> clusterState)
```
    Given the cluster state this will try and approach an optimal balance. This should always approach the optimal state given enough steps.
    
    Returns:
    
    List of plans
  - updateStochasticCosts
```
private void updateStochasticCosts(TableName tableName,
                                   Double overall,
                                   Double[] subCosts)
```
    update costs to JMX
  - functionCost
```
private String functionCost()
```
  - createRegionPlans
```
private List<RegionPlan> createRegionPlans(BaseLoadBalancer.Cluster cluster)
```
    Create all of the RegionPlan's needed to move from the initial cluster state to the desired state.
    
    Parameters:
    
    cluster - The state of the cluster
    
    Returns:
    
    List of RegionPlan's that represent the moves needed to get to desired final state.
  - updateRegionLoad
```
private void updateRegionLoad()
```
    Store the current region loads.
  - initCosts
```
protected void initCosts(BaseLoadBalancer.Cluster cluster)
```
  - updateCostsWithAction
```
protected void updateCostsWithAction(BaseLoadBalancer.Cluster cluster,
                                     BaseLoadBalancer.Cluster.Action action)
```
  - getCostFunctionNames
```
public String[] getCostFunctionNames()
```
    Get the names of the cost functions
  - computeCost
```
protected double computeCost(BaseLoadBalancer.Cluster cluster,
                             double previousCost)
```
    This is the main cost function. It will compute a cost associated with a proposed cluster state. All different costs will be combined with their multipliers to produce a double cost.
    
    Parameters:
    
    cluster - The state of the cluster
    
    previousCost - the previous cost. This is used as an early out.
    
    Returns:
    
    a double of a cost associated with the proposed cluster state. This cost is an aggregate of all individual cost functions.
  - composeAttributeName
```
public static String composeAttributeName(String tableName,
                                          String costFunctionName)
```
    A helper function to compose the attribute name from tablename and costfunction name

Class StochasticLoadBalancer

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer

Field Summary

Fields inherited from class org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer

Fields inherited from interface org.apache.hadoop.hbase.master.LoadBalancer

Constructor Summary

Method Summary

Methods inherited from class org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer

Methods inherited from class java.lang.Object

Methods inherited from interface org.apache.hadoop.hbase.master.LoadBalancer

Field Detail

STEPS_PER_REGION_KEY

MAX_STEPS_KEY

RUN_MAX_STEPS_KEY

MAX_RUNNING_TIME_KEY

KEEP_REGION_LOADS

TABLE_FUNCTION_SEP

MIN_COST_NEED_BALANCE_KEY

RANDOM

LOG

loads

maxSteps

runMaxSteps

stepsPerRegion

maxRunningTime

numRegionLoadsToRemember

minCostNeedBalance

candidateGenerators

regionLoadFunctions

costFunctions

curOverallCost

tempFunctionCosts

curFunctionCosts

localityCandidateGenerator

localityCost

rackLocalityCost

regionReplicaHostCostFunction

regionReplicaRackCostFunction

isByTable

tableName

Constructor Detail

StochasticLoadBalancer

Method Detail

onConfigurationChange

setConf

setCandidateGenerators

setSlop

setClusterMetrics

updateMetricsSize

setMasterServices

areSomeRegionReplicasColocated

needsBalance

balanceCluster

nextAction

balanceCluster

updateStochasticCosts

functionCost

createRegionPlans

updateRegionLoad

initCosts

updateCostsWithAction

getCostFunctionNames

computeCost

composeAttributeName