org.apache.hadoop.hbase.master.balancer

Class HeterogeneousRegionCountCostFunction

java.lang.Object

org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer.CostFunction

org.apache.hadoop.hbase.master.balancer.HeterogeneousRegionCountCostFunction

@InterfaceAudience.Private
public class HeterogeneousRegionCountCostFunction
extends StochasticLoadBalancer.CostFunction

This is an optional Cost function designed to allow region count skew across RegionServers. A rule file is loaded from the local FS or HDFS before balancing. It contains lines of rules. A rule is composed of a regexp for hostname, and a limit. For example, we could have:

* rs[0-9] 200 * rs1[0-9] 50

RegionServers with hostname matching the first rules will have a limit of 200, and the others 50. If there's no match, a default is set. The costFunction is trying to fill all RegionServers linearly, meaning that if the global usage is at 50%, then all RegionServers should hold half of their capacity in terms of regions. In order to use this CostFunction, you need to set the following options:

• hbase.master.balancer.stochastic.additionalCostFunctions
• hbase.master.balancer.stochastic.heterogeneousRegionCountRulesFile
• hbase.master.balancer.stochastic.heterogeneousRegionCountDefault

The rule file can be located on local FS or HDFS, depending on the prefix (file//: or hdfs://).

Field Summary

Fields

Modifier and Type	Field and Description
private org.apache.hadoop.conf.Configuration	conf
private static float	DEFAULT_REGION_COUNT_SKEW_COST
private int	defaultNumberOfRegions
private static String	HBASE_MASTER_BALANCER_HETEROGENEOUS_RULES_DEFAULT Default rule to apply when the rule file is not found.
(package private) static String	HBASE_MASTER_BALANCER_HETEROGENEOUS_RULES_FILE configuration used for the path where the rule file is stored.
private Map<ServerName,Integer>	limitPerRS This is a cache, used to not go through all the limitPerRule map when searching for limit
private Map<Pattern,Integer>	limitPerRule Contains the rules, key is the regexp for ServerName, value is the limit
private static org.slf4j.Logger	LOG
(package private) double	overallUsage
private static String	REGION_COUNT_SKEW_COST_KEY Cost for the function.
private String	rulesPath
private int	totalCapacity Total capacity of regions for the cluster, based on the online RS and their associated rules

Fields inherited from class org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer.CostFunction

cluster

Constructor Summary

Constructors

Constructor and Description
HeterogeneousRegionCountCostFunction(org.apache.hadoop.conf.Configuration conf)

Method Summary

Modifier and Type	Method and Description
protected double	cost()
(package private) int	findLimitForRS(ServerName serverName) Find the limit for a ServerName.
(package private) int	getNumberOfRulesLoaded()
(package private) void	init(BaseLoadBalancer.Cluster cluster) Called once per LB invocation to give the cost function to initialize it's state, and perform any costly calculation.
(package private) void	loadRules() used to load the rule files.
private List<String>	readFile(String filename) used to read the rule files from either HDFS or local FS
private List<String>	readFileFromHDFS(String filename) used to read the rule files from HDFS
private List<String>	readFileFromLocalFS(String filename) used to read the rule files from local FS
private List<String>	readLines(BufferedReader reader)
private void	rebuildCache() Rebuild cache matching ServerNames and their capacity.

Methods inherited from class org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer.CostFunction

costFromArray, getMultiplier, isNeeded, postAction, regionMoved, scale, setMultiplier

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

HBASE_MASTER_BALANCER_HETEROGENEOUS_RULES_FILE

static final String HBASE_MASTER_BALANCER_HETEROGENEOUS_RULES_FILE

configuration used for the path where the rule file is stored.

See Also:

Constant Field Values

LOG

private static final org.slf4j.Logger LOG

HBASE_MASTER_BALANCER_HETEROGENEOUS_RULES_DEFAULT

private static final String HBASE_MASTER_BALANCER_HETEROGENEOUS_RULES_DEFAULT

Default rule to apply when the rule file is not found. Default to 200.

See Also:

Constant Field Values

REGION_COUNT_SKEW_COST_KEY

private static final String REGION_COUNT_SKEW_COST_KEY

Cost for the function. Default to 500, can be changed.

See Also:

Constant Field Values

DEFAULT_REGION_COUNT_SKEW_COST

private static final float DEFAULT_REGION_COUNT_SKEW_COST

See Also:

Constant Field Values

rulesPath

private final String rulesPath

limitPerRule

private final Map<Pattern,Integer> limitPerRule

Contains the rules, key is the regexp for ServerName, value is the limit

limitPerRS

private final Map<ServerName,Integer> limitPerRS

This is a cache, used to not go through all the limitPerRule map when searching for limit

conf

private final org.apache.hadoop.conf.Configuration conf

defaultNumberOfRegions

private int defaultNumberOfRegions

totalCapacity

private int totalCapacity

Total capacity of regions for the cluster, based on the online RS and their associated rules

overallUsage

double overallUsage

Constructor Detail

HeterogeneousRegionCountCostFunction

HeterogeneousRegionCountCostFunction(org.apache.hadoop.conf.Configuration conf)

Method Detail

init

void init(BaseLoadBalancer.Cluster cluster)

Called once per LB invocation to give the cost function to initialize it's state, and perform any costly calculation.

Overrides:

init in class StochasticLoadBalancer.CostFunction

cost

protected double cost()

Specified by:

cost in class StochasticLoadBalancer.CostFunction

loadRules

void loadRules()

used to load the rule files.

readFile

private List<String> readFile(String filename)

used to read the rule files from either HDFS or local FS

readFileFromHDFS

private List<String> readFileFromHDFS(String filename)
                               throws IOException

used to read the rule files from HDFS

Throws:

IOException

readFileFromLocalFS

private List<String> readFileFromLocalFS(String filename)
                                  throws IOException

used to read the rule files from local FS

Throws:

IOException

readLines

private List<String> readLines(BufferedReader reader)
                        throws IOException

Throws:

IOException

rebuildCache

private void rebuildCache()

Rebuild cache matching ServerNames and their capacity.

findLimitForRS

int findLimitForRS(ServerName serverName)

Find the limit for a ServerName. If not found then return the default value

Parameters:

serverName - the server we are looking for

Returns:

the limit

getNumberOfRulesLoaded

int getNumberOfRulesLoaded()