Package org.apache.hadoop.hbase.master

Class ServerManager

java.lang.Object

org.apache.hadoop.hbase.master.ServerManager

All Implemented Interfaces:

ConfigurationObserver

@Private
public class ServerManager
extends Object
implements ConfigurationObserver

The ServerManager class manages info about region servers.

Maintains lists of online and dead servers. Processes the startups, shutdowns, and deaths of region servers.

Servers are distinguished in two different ways. A given server has a location, specified by hostname and port, and of which there can only be one online at any given time. A server instance is specified by the location (hostname and port) as well as the startcode (timestamp from when the server was started). This is used to differentiate a restarted instance of a given server from the original instance.

If a sever is known not to be running any more, it is called dead. The dead server needs to be handled by a ServerShutdownHandler. If the handler is not enabled yet, the server can't be handled right away so it is queued up. After the handler is enabled, the server will be submitted to a handler to handle. However, the handler may be just partially enabled. If so, the server cannot be fully processed, and be queued up for further processing. A server is fully processed only after the handler is fully enabled and has completed the handling.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
private class	ServerManager.FlushedSequenceIdFlusher
static enum	ServerManager.ServerLiveState

Field Summary

Fields

Modifier and Type	Field	Description
private AtomicBoolean	clusterShutdown
private final DeadServer	deadservers
private final ArrayList<ServerName>	drainingServers	List of region servers that should not get any more new regions.
private ServerManager.FlushedSequenceIdFlusher	flushedSeqIdFlusher
static final String	FLUSHEDSEQUENCEID_FLUSHER_INTERVAL
static final int	FLUSHEDSEQUENCEID_FLUSHER_INTERVAL_DEFAULT
private final ConcurrentNavigableMap<byte[],Long>	flushedSequenceIdByRegion	The last flushed sequence id for a region.
private boolean	isFlushSeqIdPersistInProgress
private static final String	LAST_FLUSHED_SEQ_ID_FILE	File on hdfs to store last flushed sequence id of regions
private List<ServerListener>	listeners	Listeners that are called on server events.
private static final org.slf4j.Logger	LOG
private final MasterServices	master
static final String	MAX_CLOCK_SKEW_MS
private final long	maxSkew
private final ConcurrentNavigableMap<ServerName,ServerMetrics>	onlineServers	Map of registered servers to their current load
static final String	PERSIST_FLUSHEDSEQUENCEID	see HBASE-20727 if set to true, flushedSequenceIdByRegion and storeFlushedSequenceIdsByRegion will be persisted to HDFS and loaded when master restart to speed up log split
static final boolean	PERSIST_FLUSHEDSEQUENCEID_DEFAULT
private boolean	persistFlushedSequenceId
private boolean	rejectDecommissionedHostsConfig	Configured value of HConstants.REJECT_DECOMMISSIONED_HOSTS_KEY
private final RegionServerList	storage
private final ConcurrentNavigableMap<byte[],ConcurrentNavigableMap<byte[],Long>>	storeFlushedSequenceIdsByRegion	The last flushed sequence id for a store in a region.
static final String	WAIT_ON_REGIONSERVERS_INTERVAL
static final String	WAIT_ON_REGIONSERVERS_MAXTOSTART
static final String	WAIT_ON_REGIONSERVERS_MINTOSTART
static final String	WAIT_ON_REGIONSERVERS_TIMEOUT
private final long	warningSkew

Constructor Summary

Constructors

Constructor	Description
ServerManager(MasterServices master, RegionServerList storage)	Constructor.

Method Summary

Modifier and Type	Method	Description
boolean	addServerToDrainList(ServerName sn)	Add the server to the drain list.
boolean	areDeadServersInProgress()	Checks if any dead servers are currently in progress.
(package private) boolean	checkAndRecordNewServer(ServerName serverName, ServerMetrics sl)	Check is a server of same host and port already exists, if not, or the existed one got a smaller start code, record it.
private void	checkClockSkew(ServerName serverName, long serverCurrentTime)	Checks if the clock skew between the server and the master.
private void	checkIsDead(ServerName serverName, String what)	Called when RegionServer first reports in for duty and thereafter each time it heartbeats to make sure it is has not been figured for dead.
private void	checkRejectableDecommissionedStatus(ServerName sn)	Checks if the Master is configured to reject decommissioned hosts or not.
(package private) void	clearDeadServersWithSameHostNameAndPortOfOnlineServer()	To clear any dead server with same host name and port of any online server
static void	closeRegionSilentlyAndWait(AsyncClusterConnection connection, ServerName server, RegionInfo region, long timeout)	Contacts a region server and waits up to timeout ms to close the region.
int	countOfRegionServers()	Returns the count of active regionservers
List<ServerName>	createDestinationServersList()	Calls createDestinationServersList(java.util.List<org.apache.hadoop.hbase.ServerName>) without server to exclude.
List<ServerName>	createDestinationServersList(List<ServerName> serversToExclude)	Creates a list of possible destinations for a region.
long	expireServer(ServerName serverName)	Expire the passed server.
(package private) long	expireServer(ServerName serverName, boolean force)
(package private) void	findDeadServersAndProcess(Set<ServerName> deadServersFromPE, Set<ServerName> liveServersFromWALDir)	Find out the region servers crashed between the crash of the previous master instance and the current master instance and schedule SCP for them.
ServerName	findServerWithSameHostnamePortWithLock(ServerName serverName)	Assumes onlineServers is locked.
double	getAverageLoad()	Compute the average load across all region servers.
DeadServer	getDeadServers()
List<ServerName>	getDrainingServersList()	Returns A copy of the internal list of draining servers.
ConcurrentNavigableMap<byte[],Long>	getFlushedSequenceIdByRegion()
int	getInfoPort(ServerName serverName)
org.apache.hadoop.hbase.shaded.protobuf.generated.ClusterStatusProtos.RegionStoreSequenceIds	getLastFlushedSequenceId(byte[] encodedRegionName)
ServerMetrics	getLoad(ServerName serverName)	Returns ServerMetrics if serverName is known else null
private int	getMinToStart()	Calculate min necessary to start.
Map<ServerName,ServerMetrics>	getOnlineServers()	Returns Read-only map of servers to serverinfo
List<ServerName>	getOnlineServersList()	Returns A copy of the internal list of online servers.
List<ServerName>	getOnlineServersListWithPredicator(List<ServerName> keys, Predicate<ServerMetrics> idleServerPredicator)
private List<String>	getRegionServersInZK(ZKWatcher zkw)
boolean	getRejectDecommissionedHostsConfig(org.apache.hadoop.conf.Configuration conf)	Reads the value of HConstants.REJECT_DECOMMISSIONED_HOSTS_KEY from the config and returns it
private String	getStrForMax(int max)
String	getVersion(ServerName serverName)	May return "0.0.0" when server is not online
int	getVersionNumber(ServerName serverName)	May return 0 when server is not online.
boolean	isClusterShutdown()
boolean	isRegionInServerManagerStates(RegionInfo hri)
boolean	isServerDead(ServerName serverName)	Check if a server is known to be dead.
ServerManager.ServerLiveState	isServerKnownAndOnline(ServerName serverName)	Returns whether the server is online, dead, or unknown.
boolean	isServerOnline(ServerName serverName)
boolean	isServerUnknown(ServerName serverName)	Check if a server is unknown.
(package private) void	letRegionServersShutdown()
void	loadLastFlushedSequenceIds()	Load last flushed sequence id of each region from HDFS, if persisted
void	moveFromOnlineToDeadServers(ServerName sn)	Called when server has expired.
void	onConfigurationChange(org.apache.hadoop.conf.Configuration conf)	Implementation of the ConfigurationObserver interface.
private void	persistRegionLastFlushedSequenceIds()	Persist last flushed sequence id of each region to HDFS
(package private) void	recordNewServerWithLock(ServerName serverName, ServerMetrics sl)	Adds the onlineServers list.
void	regionServerReport(ServerName sn, ServerMetrics sl)
(package private) ServerName	regionServerStartup(org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos.RegionServerStartupRequest request, int versionNumber, String version, InetAddress ia)	Let the server manager know a new regionserver has come online
void	registerListener(ServerListener listener)	Add the listener to the notification list.
void	removeDeletedRegionFromLoadedFlushedSequenceIds()	Regions may have been removed between latest persist of FlushedSequenceIds and master abort.
void	removeRegion(RegionInfo regionInfo)	Called by delete table and similar to notify the ServerManager that a region was removed.
void	removeRegions(List<RegionInfo> regions)	Called by delete table and similar to notify the ServerManager that a region was removed.
boolean	removeServerFromDrainList(ServerName sn)
void	shutdownCluster()
void	startChore()	start chore in ServerManager
void	stop()	Stop the ServerManager.
boolean	unregisterListener(ServerListener listener)	Remove the listener from the notification list.
private void	updateLastFlushedSequenceIds(ServerName sn, ServerMetrics hsl)	Updates last flushed sequence Ids for the regions on server sn
void	waitForRegionServers(MonitoredTask status)	Wait for the region servers to report in.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

WAIT_ON_REGIONSERVERS_MAXTOSTART

public static final String WAIT_ON_REGIONSERVERS_MAXTOSTART

See Also:

• Constant Field Values

WAIT_ON_REGIONSERVERS_MINTOSTART

public static final String WAIT_ON_REGIONSERVERS_MINTOSTART

See Also:

• Constant Field Values

WAIT_ON_REGIONSERVERS_TIMEOUT

public static final String WAIT_ON_REGIONSERVERS_TIMEOUT

See Also:

• Constant Field Values

WAIT_ON_REGIONSERVERS_INTERVAL

public static final String WAIT_ON_REGIONSERVERS_INTERVAL

See Also:

• Constant Field Values

PERSIST_FLUSHEDSEQUENCEID

public static final String PERSIST_FLUSHEDSEQUENCEID

see HBASE-20727 if set to true, flushedSequenceIdByRegion and storeFlushedSequenceIdsByRegion will be persisted to HDFS and loaded when master restart to speed up log split

See Also:

• Constant Field Values

PERSIST_FLUSHEDSEQUENCEID_DEFAULT

public static final boolean PERSIST_FLUSHEDSEQUENCEID_DEFAULT

See Also:

• Constant Field Values

FLUSHEDSEQUENCEID_FLUSHER_INTERVAL

public static final String FLUSHEDSEQUENCEID_FLUSHER_INTERVAL

See Also:

• Constant Field Values

FLUSHEDSEQUENCEID_FLUSHER_INTERVAL_DEFAULT

public static final int FLUSHEDSEQUENCEID_FLUSHER_INTERVAL_DEFAULT

See Also:

• Constant Field Values

MAX_CLOCK_SKEW_MS

public static final String MAX_CLOCK_SKEW_MS

See Also:

• Constant Field Values

LOG

private static final org.slf4j.Logger LOG

clusterShutdown

private AtomicBoolean clusterShutdown

flushedSequenceIdByRegion

private final ConcurrentNavigableMap<byte[],Long> flushedSequenceIdByRegion

The last flushed sequence id for a region.

persistFlushedSequenceId

private boolean persistFlushedSequenceId

isFlushSeqIdPersistInProgress

private volatile boolean isFlushSeqIdPersistInProgress

LAST_FLUSHED_SEQ_ID_FILE

private static final String LAST_FLUSHED_SEQ_ID_FILE

File on hdfs to store last flushed sequence id of regions

See Also:

• Constant Field Values

flushedSeqIdFlusher

private ServerManager.FlushedSequenceIdFlusher flushedSeqIdFlusher

storeFlushedSequenceIdsByRegion

private final ConcurrentNavigableMap<byte[],ConcurrentNavigableMap<byte[],Long>> storeFlushedSequenceIdsByRegion

The last flushed sequence id for a store in a region.

onlineServers

private final ConcurrentNavigableMap<ServerName,ServerMetrics> onlineServers

Map of registered servers to their current load

drainingServers

private final ArrayList<ServerName> drainingServers

List of region servers that should not get any more new regions.

master

private final MasterServices master

storage

private final RegionServerList storage

deadservers

private final DeadServer deadservers

maxSkew

private final long maxSkew

warningSkew

private final long warningSkew

listeners

private List<ServerListener> listeners

Listeners that are called on server events.

rejectDecommissionedHostsConfig

private volatile boolean rejectDecommissionedHostsConfig

Configured value of HConstants.REJECT_DECOMMISSIONED_HOSTS_KEY

Constructor Details

ServerManager

public ServerManager(MasterServices master,
 RegionServerList storage)

Constructor.

Method Details

onConfigurationChange

public void onConfigurationChange(org.apache.hadoop.conf.Configuration conf)

Implementation of the ConfigurationObserver interface. We are interested in live-loading the configuration value of HConstants.REJECT_DECOMMISSIONED_HOSTS_KEY

Specified by:

onConfigurationChange in interface ConfigurationObserver

Parameters:

conf - Server configuration instance

getRejectDecommissionedHostsConfig

public boolean getRejectDecommissionedHostsConfig(org.apache.hadoop.conf.Configuration conf)

Reads the value of HConstants.REJECT_DECOMMISSIONED_HOSTS_KEY from the config and returns it

Parameters:

conf - Configuration instance of the Master

registerListener

public void registerListener(ServerListener listener)

Add the listener to the notification list.

Parameters:

listener - The ServerListener to register

unregisterListener

public boolean unregisterListener(ServerListener listener)

Remove the listener from the notification list.

Parameters:

listener - The ServerListener to unregister

regionServerStartup

ServerName regionServerStartup(org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos.RegionServerStartupRequest request,
 int versionNumber,
 String version,
 InetAddress ia)
                        throws IOException

Let the server manager know a new regionserver has come online

Parameters:

request - the startup request

versionNumber - the version number of the new regionserver

version - the version of the new regionserver, could contain strings like "SNAPSHOT"

ia - the InetAddress from which request is received

Returns:

The ServerName we know this server as.

Throws:

IOException

updateLastFlushedSequenceIds

private void updateLastFlushedSequenceIds(ServerName sn,
 ServerMetrics hsl)

Updates last flushed sequence Ids for the regions on server sn

regionServerReport

public void regionServerReport(ServerName sn,
 ServerMetrics sl)
                        throws YouAreDeadException

Throws:

YouAreDeadException

checkRejectableDecommissionedStatus

private void checkRejectableDecommissionedStatus(ServerName sn)
                                          throws DecommissionedHostRejectedException

Checks if the Master is configured to reject decommissioned hosts or not. When it's configured to do so, any RegionServer trying to join the cluster will have it's host checked against the list of hosts of currently decommissioned servers and potentially get prevented from reporting for duty; otherwise, we do nothing and we let them pass to the next check. See HBASE-28342 for details.

Parameters:

sn - The ServerName to check for

Throws:

DecommissionedHostRejectedException - if the Master is configured to reject decommissioned hosts and this host exists in the list of the decommissioned servers

checkAndRecordNewServer

boolean checkAndRecordNewServer(ServerName serverName,
 ServerMetrics sl)

Check is a server of same host and port already exists, if not, or the existed one got a smaller start code, record it.

Parameters:

serverName - the server to check and record

sl - the server load on the server

Returns:

true if the server is recorded, otherwise, false

findDeadServersAndProcess

void findDeadServersAndProcess(Set<ServerName> deadServersFromPE,
 Set<ServerName> liveServersFromWALDir)

Find out the region servers crashed between the crash of the previous master instance and the current master instance and schedule SCP for them.

Since the RegionServerTracker has already helped us to construct the online servers set by scanning zookeeper, now we can compare the online servers with liveServersFromWALDir to find out whether there are servers which are already dead.

Must be called inside the initialization method of RegionServerTracker to avoid concurrency issue.

Parameters:

deadServersFromPE - the region servers which already have a SCP associated.

liveServersFromWALDir - the live region servers from wal directory.

checkClockSkew

private void checkClockSkew(ServerName serverName,
 long serverCurrentTime)
                     throws ClockOutOfSyncException

Checks if the clock skew between the server and the master. If the clock skew exceeds the configured max, it will throw an exception; if it exceeds the configured warning threshold, it will log a warning but start normally.

Parameters:

serverName - Incoming servers's name

Throws:

ClockOutOfSyncException - if the skew exceeds the configured max value

checkIsDead

private void checkIsDead(ServerName serverName,
 String what)
                  throws YouAreDeadException

Called when RegionServer first reports in for duty and thereafter each time it heartbeats to make sure it is has not been figured for dead. If this server is on the dead list, reject it with a YouAreDeadException. If it was dead but came back with a new start code, remove the old entry from the dead list.

Parameters:

what - START or REPORT

Throws:

YouAreDeadException

findServerWithSameHostnamePortWithLock

public ServerName findServerWithSameHostnamePortWithLock(ServerName serverName)

Assumes onlineServers is locked.

Returns:

ServerName with matching hostname and port.

recordNewServerWithLock

void recordNewServerWithLock(ServerName serverName,
 ServerMetrics sl)

Adds the onlineServers list. onlineServers should be locked.

Parameters:

serverName - The remote servers name.

getFlushedSequenceIdByRegion

public ConcurrentNavigableMap<byte[],Long> getFlushedSequenceIdByRegion()

getLastFlushedSequenceId

public org.apache.hadoop.hbase.shaded.protobuf.generated.ClusterStatusProtos.RegionStoreSequenceIds getLastFlushedSequenceId(byte[] encodedRegionName)

getLoad

public ServerMetrics getLoad(ServerName serverName)

Returns ServerMetrics if serverName is known else null

getAverageLoad

public double getAverageLoad()

Compute the average load across all region servers. Currently, this uses a very naive computation - just uses the number of regions being served, ignoring stats about number of requests.

Returns:

the average load

countOfRegionServers

public int countOfRegionServers()

Returns the count of active regionservers

getOnlineServers

public Map<ServerName,ServerMetrics> getOnlineServers()

Returns Read-only map of servers to serverinfo

getDeadServers

public DeadServer getDeadServers()

areDeadServersInProgress

public boolean areDeadServersInProgress()
                                 throws IOException

Checks if any dead servers are currently in progress.

Returns:

true if any RS are being processed as dead, false if not

Throws:

IOException

letRegionServersShutdown

void letRegionServersShutdown()

getRegionServersInZK

private List<String> getRegionServersInZK(ZKWatcher zkw)
                                   throws org.apache.zookeeper.KeeperException

Throws:

org.apache.zookeeper.KeeperException

expireServer

public long expireServer(ServerName serverName)

Expire the passed server. Add it to list of dead servers and queue a shutdown processing.

Returns:

pid if we queued a ServerCrashProcedure else Procedure.NO_PROC_ID if we did not (could happen for many reasons including the fact that its this server that is going down or we already have queued an SCP for this server or SCP processing is currently disabled because we are in startup phase).

expireServer

long expireServer(ServerName serverName,
 boolean force)

moveFromOnlineToDeadServers

public void moveFromOnlineToDeadServers(ServerName sn)

Called when server has expired.

removeServerFromDrainList

public boolean removeServerFromDrainList(ServerName sn)

addServerToDrainList

public boolean addServerToDrainList(ServerName sn)

Add the server to the drain list.

Returns:

True if the server is added or the server is already on the drain list.

closeRegionSilentlyAndWait

public static void closeRegionSilentlyAndWait(AsyncClusterConnection connection,
 ServerName server,
 RegionInfo region,
 long timeout)
                                       throws IOException,
InterruptedException

Contacts a region server and waits up to timeout ms to close the region. This bypasses the active hmaster. Pass -1 as timeout if you do not want to wait on result.

Throws:

IOException

InterruptedException

getMinToStart

private int getMinToStart()

Calculate min necessary to start. This is not an absolute. It is just a friction that will cause us hang around a bit longer waiting on RegionServers to check-in.

waitForRegionServers

public void waitForRegionServers(MonitoredTask status)
                          throws InterruptedException

Wait for the region servers to report in. We will wait until one of this condition is met: - the master is stopped - the 'hbase.master.wait.on.regionservers.maxtostart' number of region servers is reached - the 'hbase.master.wait.on.regionservers.mintostart' is reached AND there have been no new region server in for 'hbase.master.wait.on.regionservers.interval' time AND the 'hbase.master.wait.on.regionservers.timeout' is reached

Throws:

InterruptedException

getStrForMax

private String getStrForMax(int max)

getOnlineServersList

public List<ServerName> getOnlineServersList()

Returns A copy of the internal list of online servers.

getOnlineServersListWithPredicator

public List<ServerName> getOnlineServersListWithPredicator(List<ServerName> keys,
 Predicate<ServerMetrics> idleServerPredicator)

Parameters:

keys - The target server name

idleServerPredicator - Evaluates the server on the given load

Returns:

A copy of the internal list of online servers matched by the predicator

getDrainingServersList

public List<ServerName> getDrainingServersList()

Returns A copy of the internal list of draining servers.

isServerOnline

public boolean isServerOnline(ServerName serverName)

isServerKnownAndOnline

public ServerManager.ServerLiveState isServerKnownAndOnline(ServerName serverName)

Returns whether the server is online, dead, or unknown.

isServerDead

public boolean isServerDead(ServerName serverName)

Check if a server is known to be dead. A server can be online, or known to be dead, or unknown to this manager (i.e, not online, not known to be dead either; it is simply not tracked by the master any more, for example, a very old previous instance).

isServerUnknown

public boolean isServerUnknown(ServerName serverName)

Check if a server is unknown. A server can be online, or known to be dead, or unknown to this manager (i.e, not online, not known to be dead either; it is simply not tracked by the master any more, for example, a very old previous instance).

shutdownCluster

public void shutdownCluster()

isClusterShutdown

public boolean isClusterShutdown()

startChore

public void startChore()

start chore in ServerManager

stop

public void stop()

Stop the ServerManager.

createDestinationServersList

public List<ServerName> createDestinationServersList(List<ServerName> serversToExclude)

Creates a list of possible destinations for a region. It contains the online servers, but not the draining or dying servers.

Parameters:

serversToExclude - can be null if there is no server to exclude

createDestinationServersList

public List<ServerName> createDestinationServersList()

Calls createDestinationServersList(java.util.List<org.apache.hadoop.hbase.ServerName>) without server to exclude.

clearDeadServersWithSameHostNameAndPortOfOnlineServer

void clearDeadServersWithSameHostNameAndPortOfOnlineServer()

To clear any dead server with same host name and port of any online server

removeRegion

public void removeRegion(RegionInfo regionInfo)

Called by delete table and similar to notify the ServerManager that a region was removed.

isRegionInServerManagerStates

public boolean isRegionInServerManagerStates(RegionInfo hri)

removeRegions

public void removeRegions(List<RegionInfo> regions)

Called by delete table and similar to notify the ServerManager that a region was removed.

getVersionNumber

public int getVersionNumber(ServerName serverName)

May return 0 when server is not online.

getVersion

public String getVersion(ServerName serverName)

May return "0.0.0" when server is not online

getInfoPort

public int getInfoPort(ServerName serverName)

persistRegionLastFlushedSequenceIds

private void persistRegionLastFlushedSequenceIds()
                                          throws IOException

Persist last flushed sequence id of each region to HDFS

Throws:

IOException - if persit to HDFS fails

loadLastFlushedSequenceIds

public void loadLastFlushedSequenceIds()
                                throws IOException

Load last flushed sequence id of each region from HDFS, if persisted

Throws:

IOException

removeDeletedRegionFromLoadedFlushedSequenceIds

public void removeDeletedRegionFromLoadedFlushedSequenceIds()

Regions may have been removed between latest persist of FlushedSequenceIds and master abort. So after loading FlushedSequenceIds from file, and after meta loaded, we need to remove the deleted region according to RegionStates.