Package org.apache.hadoop.hbase.master.balancer

Class HeterogeneousRegionCountCostFunction

java.lang.Object

org.apache.hadoop.hbase.master.balancer.CostFunction

org.apache.hadoop.hbase.master.balancer.HeterogeneousRegionCountCostFunction

@Private
public class HeterogeneousRegionCountCostFunction
extends CostFunction

This is an optional Cost function designed to allow region count skew across RegionServers. A rule file is loaded from the local FS or HDFS before balancing. It contains lines of rules. A rule is composed of a regexp for hostname, and a limit. For example, we could have:

* rs[0-9] 200 * rs1[0-9] 50

RegionServers with hostname matching the first rules will have a limit of 200, and the others 50. If there's no match, a default is set. The costFunction is trying to fill all RegionServers linearly, meaning that if the global usage is at 50%, then all RegionServers should hold half of their capacity in terms of regions. In order to use this CostFunction, you need to set the following options:

• hbase.master.balancer.stochastic.additionalCostFunctions
• hbase.master.balancer.stochastic.heterogeneousRegionCountRulesFile
• hbase.master.balancer.stochastic.heterogeneousRegionCountDefault

The rule file can be located on local FS or HDFS, depending on the prefix (file//: or hdfs://).

Field Summary

Fields

Modifier and Type	Field	Description
private final org.apache.hadoop.conf.Configuration	conf
private static final float	DEFAULT_REGION_COUNT_SKEW_COST
private int	defaultNumberOfRegions
private static final String	HBASE_MASTER_BALANCER_HETEROGENEOUS_RULES_DEFAULT	Default rule to apply when the rule file is not found.
(package private) static final String	HBASE_MASTER_BALANCER_HETEROGENEOUS_RULES_FILE	configuration used for the path where the rule file is stored.
private final Map<ServerName,Integer>	limitPerRS	This is a cache, used to not go through all the limitPerRule map when searching for limit
private final Map<Pattern,Integer>	limitPerRule	Contains the rules, key is the regexp for ServerName, value is the limit
private static final org.slf4j.Logger	LOG
(package private) double	overallUsage
private static final String	REGION_COUNT_SKEW_COST_KEY	Cost for the function.
private final String	rulesPath
private int	totalCapacity	Total capacity of regions for the cluster, based on the online RS and their associated rules

Fields inherited from class org.apache.hadoop.hbase.master.balancer.CostFunction

cluster, COST_EPSILON

Constructor Summary

Constructors

Constructor	Description
HeterogeneousRegionCountCostFunction(org.apache.hadoop.conf.Configuration conf)

Method Summary

Modifier and Type	Method	Description
protected double	cost()
(package private) int	findLimitForRS(ServerName serverName)	Find the limit for a ServerName.
(package private) int	getNumberOfRulesLoaded()
(package private) void	loadRules()	used to load the rule files.
(package private) void	prepare(BalancerClusterState cluster)	Called once per LB invocation to give the cost function to initialize it's state, and perform any costly calculation.
private List<String>	readFile(String filename)	used to read the rule files from either HDFS or local FS
private List<String>	readFileFromHDFS(String filename)	used to read the rule files from HDFS
private List<String>	readFileFromLocalFS(String filename)	used to read the rule files from local FS
private List<String>	readLines(BufferedReader reader)
private void	rebuildCache()	Rebuild cache matching ServerNames and their capacity.

Methods inherited from class org.apache.hadoop.hbase.master.balancer.CostFunction

getMultiplier, isNeeded, postAction, regionMoved, scale, setMultiplier, updateWeight

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

HBASE_MASTER_BALANCER_HETEROGENEOUS_RULES_FILE

static final String HBASE_MASTER_BALANCER_HETEROGENEOUS_RULES_FILE

configuration used for the path where the rule file is stored.

See Also:

• Constant Field Values

LOG

private static final org.slf4j.Logger LOG

HBASE_MASTER_BALANCER_HETEROGENEOUS_RULES_DEFAULT

private static final String HBASE_MASTER_BALANCER_HETEROGENEOUS_RULES_DEFAULT

Default rule to apply when the rule file is not found. Default to 200.

See Also:

• Constant Field Values

REGION_COUNT_SKEW_COST_KEY

private static final String REGION_COUNT_SKEW_COST_KEY

Cost for the function. Default to 500, can be changed.

See Also:

• Constant Field Values

DEFAULT_REGION_COUNT_SKEW_COST

private static final float DEFAULT_REGION_COUNT_SKEW_COST

See Also:

• Constant Field Values

rulesPath

private final String rulesPath

limitPerRule

private final Map<Pattern,Integer> limitPerRule

Contains the rules, key is the regexp for ServerName, value is the limit

limitPerRS

private final Map<ServerName,Integer> limitPerRS

This is a cache, used to not go through all the limitPerRule map when searching for limit

conf

private final org.apache.hadoop.conf.Configuration conf

defaultNumberOfRegions

private int defaultNumberOfRegions

totalCapacity

private int totalCapacity

Total capacity of regions for the cluster, based on the online RS and their associated rules

overallUsage

double overallUsage

Constructor Details

HeterogeneousRegionCountCostFunction

HeterogeneousRegionCountCostFunction(org.apache.hadoop.conf.Configuration conf)

Method Details

prepare

void prepare(BalancerClusterState cluster)

Called once per LB invocation to give the cost function to initialize it's state, and perform any costly calculation.

Overrides:

prepare in class CostFunction

cost

protected double cost()

Specified by:

cost in class CostFunction

loadRules

void loadRules()

used to load the rule files.

readFile

private List<String> readFile(String filename)

used to read the rule files from either HDFS or local FS

readFileFromHDFS

private List<String> readFileFromHDFS(String filename)
                               throws IOException

used to read the rule files from HDFS

Throws:

IOException

readFileFromLocalFS

private List<String> readFileFromLocalFS(String filename)
                                  throws IOException

used to read the rule files from local FS

Throws:

IOException

readLines

private List<String> readLines(BufferedReader reader)
                        throws IOException

Throws:

IOException

rebuildCache

private void rebuildCache()

Rebuild cache matching ServerNames and their capacity.

findLimitForRS

int findLimitForRS(ServerName serverName)

Find the limit for a ServerName. If not found then return the default value

Parameters:

serverName - the server we are looking for

Returns:

the limit

getNumberOfRulesLoaded

int getNumberOfRulesLoaded()