org.apache.hadoop.hbase.master.balancer

Class BaseLoadBalancer

java.lang.Object

org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, ConfigurationObserver, LoadBalancer, Stoppable

Direct Known Subclasses:

FavoredNodeLoadBalancer, SimpleLoadBalancer, StochasticLoadBalancer

@InterfaceAudience.Private
public abstract class BaseLoadBalancer
extends Object
implements LoadBalancer

The base class for load balancers. It provides the the functions used to by AssignmentManager to assign regions in the edge cases. It doesn't provide an implementation of the actual balancing algorithm.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	BaseLoadBalancer.Cluster An efficient array based implementation similar to ClusterState for keeping the status of the cluster in terms of region assignment and distribution.
private static class	BaseLoadBalancer.DefaultRackManager

Field Summary

Fields

Modifier and Type	Field and Description
static String	BALANCER_DECISION_BUFFER_ENABLED
protected ClusterMetrics	clusterStatus
protected org.apache.hadoop.conf.Configuration	config
static boolean	DEFAULT_BALANCER_DECISION_BUFFER_ENABLED
private static List<RegionInfo>	EMPTY_REGION_LIST
(package private) static Predicate<ServerMetrics>	IDLE_SERVER_PREDICATOR
protected boolean	isByTable
private static org.slf4j.Logger	LOG
protected boolean	maintenanceMode
protected ServerName	masterServerName
protected MetricsBalancer	metricsBalancer
protected static int	MIN_SERVER_BALANCE
protected NamedQueueRecorder	namedQueueRecorder Use to add balancer decision history to ring-buffer
protected boolean	onlySystemTablesOnMaster Deprecated. since 2.4.0, will be removed in 3.0.0.
protected float	overallSlop
protected RackManager	rackManager
private static Random	RANDOM
protected RegionLocationFinder	regionFinder
protected MasterServices	services
protected float	slop
private boolean	stopped
protected boolean	useRegionFinder

Fields inherited from interface org.apache.hadoop.hbase.master.LoadBalancer

BOGUS_SERVER_NAME, SYSTEM_TABLES_ON_MASTER, TABLES_ON_MASTER

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	BaseLoadBalancer() The constructor that uses the basic MetricsBalancer
protected	BaseLoadBalancer(MetricsBalancer metricsBalancer) This Constructor accepts an instance of MetricsBalancer, which will be used instead of creating a new one

Method Summary

Modifier and Type	Method and Description
protected boolean	areSomeRegionReplicasColocated(BaseLoadBalancer.Cluster c) Subclasses should implement this to return true if the cluster has nodes that hosts multiple replicas for the same region, or, if there are multiple racks and the same rack hosts replicas of the same region
protected Map<ServerName,List<RegionInfo>>	assignMasterSystemRegions(Collection<RegionInfo> regions, List<ServerName> servers) Deprecated. since 2.4.0, will be removed in 3.0.0.
List<RegionPlan>	balanceCluster(Map<TableName,Map<ServerName,List<RegionInfo>>> loadOfAllTable) Perform the major balance operation for cluster, will invoke LoadBalancer.balanceTable(org.apache.hadoop.hbase.TableName, java.util.Map<org.apache.hadoop.hbase.ServerName, java.util.List<org.apache.hadoop.hbase.client.RegionInfo>>) to do actual balance.
protected List<RegionPlan>	balanceMasterRegions(Map<ServerName,List<RegionInfo>> clusterMap) Deprecated. since 2.4.0, will be removed in 3.0.0.
abstract List<RegionPlan>	balanceTable(TableName tableName, Map<ServerName,List<RegionInfo>> loadOfOneTable) Perform the major balance operation for table, all class implement of LoadBalancer should override this method
protected BaseLoadBalancer.Cluster	createCluster(List<ServerName> servers, Collection<RegionInfo> regions)
private void	createRegionFinder()
private List<ServerName>	findIdleServers(List<ServerName> servers)
org.apache.hadoop.conf.Configuration	getConf()
protected Map<ServerName,List<RegionInfo>>	getRegionAssignmentsByServer(Collection<RegionInfo> regions)
protected boolean	idleRegionServerExist(BaseLoadBalancer.Cluster c)
void	initialize() Initialize the load balancer.
boolean	isStopped()
protected boolean	needsBalance(TableName tableName, BaseLoadBalancer.Cluster c)
void	onConfigurationChange(org.apache.hadoop.conf.Configuration conf) This method would be called by the ConfigurationManager object when the Configuration object is reloaded from disk.
void	postMasterStartupInitialize() If balancer needs to do initialization after Master has started up, lets do that here.
private ServerName	randomAssignment(BaseLoadBalancer.Cluster cluster, RegionInfo regionInfo, List<ServerName> servers) Used to assign a single region to a random server.
ServerName	randomAssignment(RegionInfo regionInfo, List<ServerName> servers) Used to assign a single region to a random server.
void	regionOffline(RegionInfo regionInfo) Marks the region as offline at balancer.
void	regionOnline(RegionInfo regionInfo, ServerName sn) Marks the region as online at balancer.
Map<ServerName,List<RegionInfo>>	retainAssignment(Map<RegionInfo,ServerName> regions, List<ServerName> servers) Generates a bulk assignment startup plan, attempting to reuse the existing assignment information from META, but adjusting for the specified list of available/online servers available for assignment.
private void	roundRobinAssignment(BaseLoadBalancer.Cluster cluster, List<RegionInfo> regions, List<ServerName> servers, Map<ServerName,List<RegionInfo>> assignments) Round robin a list of regions to a list of servers
Map<ServerName,List<RegionInfo>>	roundRobinAssignment(List<RegionInfo> regions, List<ServerName> servers) Generates a bulk assignment plan to be used on cluster startup using a simple round-robin assignment.
void	setClusterMetrics(ClusterMetrics st) Set the current cluster status.
void	setConf(org.apache.hadoop.conf.Configuration conf)
void	setMasterServices(MasterServices masterServices) Set the master service.
void	setRackManager(RackManager rackManager)
protected void	setSlop(org.apache.hadoop.conf.Configuration conf)
boolean	shouldBeOnMaster(RegionInfo region) Deprecated. since 2.4.0, will be removed in 3.0.0.
void	stop(String why) Stop this service.
private Map<ServerName,List<RegionInfo>>	toEnsumbleTableLoad(Map<TableName,Map<ServerName,List<RegionInfo>>> LoadOfAllTable)
void	updateBalancerStatus(boolean status) Updates the balancer status tag reported to JMX

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.hbase.master.LoadBalancer

isMasterCanHostUserRegions, isSystemTablesOnlyOnMaster, isTablesOnMaster

Field Detail

BALANCER_DECISION_BUFFER_ENABLED

public static final String BALANCER_DECISION_BUFFER_ENABLED

See Also:

Constant Field Values

DEFAULT_BALANCER_DECISION_BUFFER_ENABLED

public static final boolean DEFAULT_BALANCER_DECISION_BUFFER_ENABLED

See Also:

Constant Field Values

MIN_SERVER_BALANCE

protected static final int MIN_SERVER_BALANCE

See Also:

Constant Field Values

stopped

private volatile boolean stopped

EMPTY_REGION_LIST

private static final List<RegionInfo> EMPTY_REGION_LIST

IDLE_SERVER_PREDICATOR

static final Predicate<ServerMetrics> IDLE_SERVER_PREDICATOR

regionFinder

protected RegionLocationFinder regionFinder

useRegionFinder

protected boolean useRegionFinder

isByTable

protected boolean isByTable

namedQueueRecorder

protected NamedQueueRecorder namedQueueRecorder

Use to add balancer decision history to ring-buffer

slop

protected float slop

overallSlop

protected float overallSlop

config

protected org.apache.hadoop.conf.Configuration config

rackManager

protected RackManager rackManager

RANDOM

private static final Random RANDOM

LOG

private static final org.slf4j.Logger LOG

metricsBalancer

protected MetricsBalancer metricsBalancer

clusterStatus

protected ClusterMetrics clusterStatus

masterServerName

protected ServerName masterServerName

services

protected MasterServices services

onlySystemTablesOnMaster

@Deprecated
protected boolean onlySystemTablesOnMaster

Deprecated. since 2.4.0, will be removed in 3.0.0.

See Also:

HBASE-15549

maintenanceMode

protected boolean maintenanceMode

Constructor Detail

BaseLoadBalancer

protected BaseLoadBalancer()

The constructor that uses the basic MetricsBalancer

BaseLoadBalancer

protected BaseLoadBalancer(MetricsBalancer metricsBalancer)

This Constructor accepts an instance of MetricsBalancer, which will be used instead of creating a new one

Method Detail

createRegionFinder

private void createRegionFinder()

setConf

public void setConf(org.apache.hadoop.conf.Configuration conf)

Specified by:

setConf in interface org.apache.hadoop.conf.Configurable

setSlop

protected void setSlop(org.apache.hadoop.conf.Configuration conf)

shouldBeOnMaster

@Deprecated
public boolean shouldBeOnMaster(RegionInfo region)

Deprecated. since 2.4.0, will be removed in 3.0.0.

Check if a region belongs to some system table. If so, the primary replica may be expected to be put on the master regionserver.

See Also:

HBASE-15549

balanceMasterRegions

@Deprecated
protected List<RegionPlan> balanceMasterRegions(Map<ServerName,List<RegionInfo>> clusterMap)

Deprecated. since 2.4.0, will be removed in 3.0.0.

Balance the regions that should be on master regionserver.

See Also:

HBASE-15549

assignMasterSystemRegions

@Deprecated
 @NonNull
protected Map<ServerName,List<RegionInfo>> assignMasterSystemRegions(Collection<RegionInfo> regions,
                                                                                           List<ServerName> servers)

Deprecated. since 2.4.0, will be removed in 3.0.0.

If master is configured to carry system tables only, in here is where we figure what to assign it.

See Also:

HBASE-15549

getConf

public org.apache.hadoop.conf.Configuration getConf()

Specified by:

getConf in interface org.apache.hadoop.conf.Configurable

setClusterMetrics

public void setClusterMetrics(ClusterMetrics st)

Description copied from interface: LoadBalancer

Set the current cluster status. This allows a LoadBalancer to map host name to a server

Specified by:

setClusterMetrics in interface LoadBalancer

setMasterServices

public void setMasterServices(MasterServices masterServices)

Description copied from interface: LoadBalancer

Set the master service.

Specified by:

setMasterServices in interface LoadBalancer

postMasterStartupInitialize

public void postMasterStartupInitialize()

Description copied from interface: LoadBalancer

If balancer needs to do initialization after Master has started up, lets do that here.

Specified by:

postMasterStartupInitialize in interface LoadBalancer

setRackManager

public void setRackManager(RackManager rackManager)

needsBalance

protected boolean needsBalance(TableName tableName,
                               BaseLoadBalancer.Cluster c)

areSomeRegionReplicasColocated

protected boolean areSomeRegionReplicasColocated(BaseLoadBalancer.Cluster c)

Subclasses should implement this to return true if the cluster has nodes that hosts multiple replicas for the same region, or, if there are multiple racks and the same rack hosts replicas of the same region

Parameters:

c - Cluster information

Returns:

whether region replicas are currently co-located

idleRegionServerExist

protected final boolean idleRegionServerExist(BaseLoadBalancer.Cluster c)

roundRobinAssignment

@NonNull
public Map<ServerName,List<RegionInfo>> roundRobinAssignment(List<RegionInfo> regions,
                                                                      List<ServerName> servers)
                                                               throws HBaseIOException

Generates a bulk assignment plan to be used on cluster startup using a simple round-robin assignment.

Takes a list of all the regions and all the servers in the cluster and returns a map of each server to the regions that it should be assigned.

Currently implemented as a round-robin assignment. Same invariant as load balancing, all servers holding floor(avg) or ceiling(avg). TODO: Use block locations from HDFS to place regions with their blocks

Specified by:

roundRobinAssignment in interface LoadBalancer

Parameters:

regions - all regions

servers - all servers

Returns:

map of server to the regions it should take, or emptyMap if no assignment is possible (ie. no servers)

Throws:

HBaseIOException

createCluster

protected BaseLoadBalancer.Cluster createCluster(List<ServerName> servers,
                                                 Collection<RegionInfo> regions)
                                          throws HBaseIOException

Throws:

HBaseIOException

findIdleServers

private List<ServerName> findIdleServers(List<ServerName> servers)

randomAssignment

public ServerName randomAssignment(RegionInfo regionInfo,
                                   List<ServerName> servers)
                            throws HBaseIOException

Used to assign a single region to a random server.

Specified by:

randomAssignment in interface LoadBalancer

Parameters:

regionInfo - Region for which this selection is being done.

Returns:

Servername

Throws:

HBaseIOException

retainAssignment

@NonNull
public Map<ServerName,List<RegionInfo>> retainAssignment(Map<RegionInfo,ServerName> regions,
                                                                  List<ServerName> servers)
                                                           throws HBaseIOException

Generates a bulk assignment startup plan, attempting to reuse the existing assignment information from META, but adjusting for the specified list of available/online servers available for assignment.

Takes a map of all regions to their existing assignment from META. Also takes a list of online servers for regions to be assigned to. Attempts to retain all assignment, so in some instances initial assignment will not be completely balanced.

Any leftover regions without an existing server to be assigned to will be assigned randomly to available servers.

Specified by:

retainAssignment in interface LoadBalancer

Parameters:

regions - regions and existing assignment from meta

servers - available servers

Returns:

map of servers and regions to be assigned to them, or emptyMap if no assignment is possible (ie. no servers)

Throws:

HBaseIOException

initialize

public void initialize()
                throws HBaseIOException

Description copied from interface: LoadBalancer

Initialize the load balancer. Must be called after setters.

Specified by:

initialize in interface LoadBalancer

Throws:

HBaseIOException

regionOnline

public void regionOnline(RegionInfo regionInfo,
                         ServerName sn)

Description copied from interface: LoadBalancer

Marks the region as online at balancer.

Specified by:

regionOnline in interface LoadBalancer

regionOffline

public void regionOffline(RegionInfo regionInfo)

Description copied from interface: LoadBalancer

Marks the region as offline at balancer.

Specified by:

regionOffline in interface LoadBalancer

isStopped

public boolean isStopped()

Specified by:

isStopped in interface Stoppable

Returns:

True if Stoppable.stop(String) has been closed.

stop

public void stop(String why)

Description copied from interface: Stoppable

Stop this service. Implementers should favor logging errors over throwing RuntimeExceptions.

Specified by:

stop in interface Stoppable

Parameters:

why - Why we're stopping.

updateBalancerStatus

public void updateBalancerStatus(boolean status)

Updates the balancer status tag reported to JMX

Specified by:

updateBalancerStatus in interface LoadBalancer

randomAssignment

private ServerName randomAssignment(BaseLoadBalancer.Cluster cluster,
                                    RegionInfo regionInfo,
                                    List<ServerName> servers)

Used to assign a single region to a random server.

roundRobinAssignment

private void roundRobinAssignment(BaseLoadBalancer.Cluster cluster,
                                  List<RegionInfo> regions,
                                  List<ServerName> servers,
                                  Map<ServerName,List<RegionInfo>> assignments)

Round robin a list of regions to a list of servers

getRegionAssignmentsByServer

protected Map<ServerName,List<RegionInfo>> getRegionAssignmentsByServer(Collection<RegionInfo> regions)

toEnsumbleTableLoad

private Map<ServerName,List<RegionInfo>> toEnsumbleTableLoad(Map<TableName,Map<ServerName,List<RegionInfo>>> LoadOfAllTable)

balanceTable

public abstract List<RegionPlan> balanceTable(TableName tableName,
                                              Map<ServerName,List<RegionInfo>> loadOfOneTable)

Description copied from interface: LoadBalancer

Perform the major balance operation for table, all class implement of LoadBalancer should override this method

Specified by:

balanceTable in interface LoadBalancer

Parameters:

tableName - the table to be balanced

loadOfOneTable - region load of servers for the specific one table

Returns:

List of plans

balanceCluster

public List<RegionPlan> balanceCluster(Map<TableName,Map<ServerName,List<RegionInfo>>> loadOfAllTable)

Description copied from interface: LoadBalancer

Perform the major balance operation for cluster, will invoke LoadBalancer.balanceTable(org.apache.hadoop.hbase.TableName, java.util.Map<org.apache.hadoop.hbase.ServerName, java.util.List<org.apache.hadoop.hbase.client.RegionInfo>>) to do actual balance. Normally not need override this method, except SimpleLoadBalancer and RSGroupBasedLoadBalancer.

Specified by:

balanceCluster in interface LoadBalancer

Parameters:

loadOfAllTable - region load of servers for all table

Returns:

a list of regions to be moved, including source and destination, or null if cluster is already balanced

onConfigurationChange

public void onConfigurationChange(org.apache.hadoop.conf.Configuration conf)

Description copied from interface: ConfigurationObserver

This method would be called by the ConfigurationManager object when the Configuration object is reloaded from disk.

Specified by:

onConfigurationChange in interface ConfigurationObserver

Specified by:

onConfigurationChange in interface LoadBalancer