org.apache.hadoop.hbase.master

Class ServerManager

java.lang.Object

org.apache.hadoop.hbase.master.ServerManager

@InterfaceAudience.Private
public class ServerManager
extends Object

The ServerManager class manages info about region servers.

Maintains lists of online and dead servers. Processes the startups, shutdowns, and deaths of region servers.

Servers are distinguished in two different ways. A given server has a location, specified by hostname and port, and of which there can only be one online at any given time. A server instance is specified by the location (hostname and port) as well as the startcode (timestamp from when the server was started). This is used to differentiate a restarted instance of a given server from the original instance.

If a sever is known not to be running any more, it is called dead. The dead server needs to be handled by a ServerShutdownHandler. If the handler is not enabled yet, the server can't be handled right away so it is queued up. After the handler is enabled, the server will be submitted to a handler to handle. However, the handler may be just partially enabled. If so, the server cannot be fully processed, and be queued up for further processing. A server is fully processed only after the handler is fully enabled and has completed the handling.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	ServerManager.ServerLiveState

Field Summary

Fields

Modifier and Type	Field and Description
private AtomicBoolean	clusterShutdown
private ClusterConnection	connection
private DeadServer	deadservers
private ArrayList<ServerName>	drainingServers List of region servers that should not get any more new regions.
private ConcurrentNavigableMap<byte[],Long>	flushedSequenceIdByRegion The last flushed sequence id for a region.
private List<ServerListener>	listeners Listeners that are called on server events.
private static org.slf4j.Logger	LOG
private MasterServices	master
private long	maxSkew
private ConcurrentNavigableMap<ServerName,ServerMetrics>	onlineServers Map of registered servers to their current load
private RpcControllerFactory	rpcControllerFactory
private Map<ServerName,org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos.AdminService.BlockingInterface>	rsAdmins Map of admin interfaces per registered regionserver; these interfaces we use to control regionservers out on the cluster
private RegionServerList	storage
private ConcurrentNavigableMap<byte[],ConcurrentNavigableMap<byte[],Long>>	storeFlushedSequenceIdsByRegion The last flushed sequence id for a store in a region.
static String	WAIT_ON_REGIONSERVERS_INTERVAL
static String	WAIT_ON_REGIONSERVERS_MAXTOSTART
static String	WAIT_ON_REGIONSERVERS_MINTOSTART
static String	WAIT_ON_REGIONSERVERS_TIMEOUT
private long	warningSkew

Constructor Summary

Constructors

Constructor and Description
ServerManager(MasterServices master, RegionServerList storage) Constructor.

Method Summary

Modifier and Type	Method and Description
boolean	addServerToDrainList(ServerName sn) Add the server to the drain list.
boolean	areDeadServersInProgress() Checks if any dead servers are currently in progress.
(package private) boolean	checkAndRecordNewServer(ServerName serverName, ServerMetrics sl) Check is a server of same host and port already exists, if not, or the existed one got a smaller start code, record it.
private void	checkClockSkew(ServerName serverName, long serverCurrentTime) Checks if the clock skew between the server and the master.
private void	checkIsDead(ServerName serverName, String what) Called when RegionServer first reports in for duty and thereafter each time it heartbeats to make sure it is has not been figured for dead.
(package private) void	clearDeadServersWithSameHostNameAndPortOfOnlineServer() To clear any dead server with same host name and port of any online server
static void	closeRegionSilentlyAndWait(ClusterConnection connection, ServerName server, RegionInfo region, long timeout) Contacts a region server and waits up to timeout ms to close the region.
int	countOfRegionServers() Returns the count of active regionservers
List<ServerName>	createDestinationServersList() Calls createDestinationServersList(java.util.List<org.apache.hadoop.hbase.ServerName>) without server to exclude.
List<ServerName>	createDestinationServersList(List<ServerName> serversToExclude) Creates a list of possible destinations for a region.
long	expireServer(ServerName serverName) Expire the passed server.
(package private) long	expireServer(ServerName serverName, boolean force)
(package private) void	findDeadServersAndProcess(Set<ServerName> deadServersFromPE, Set<ServerName> liveServersFromWALDir) Find out the region servers crashed between the crash of the previous master instance and the current master instance and schedule SCP for them.
private ServerName	findServerWithSameHostnamePortWithLock(ServerName serverName) Assumes onlineServers is locked.
double	getAverageLoad() Compute the average load across all region servers.
DeadServer	getDeadServers()
List<ServerName>	getDrainingServersList() Returns A copy of the internal list of draining servers.
int	getInfoPort(ServerName serverName)
org.apache.hadoop.hbase.shaded.protobuf.generated.ClusterStatusProtos.RegionStoreSequenceIds	getLastFlushedSequenceId(byte[] encodedRegionName)
ServerMetrics	getLoad(ServerName serverName) n * @return ServerMetrics if serverName is known else null
private int	getMinToStart() Calculate min necessary to start.
Map<ServerName,ServerMetrics>	getOnlineServers() Returns Read-only map of servers to serverinfo
List<ServerName>	getOnlineServersList() Returns A copy of the internal list of online servers.
List<ServerName>	getOnlineServersListWithPredicator(List<ServerName> keys, Predicate<ServerMetrics> idleServerPredicator)
private List<String>	getRegionServersInZK(ZKWatcher zkw)
org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos.AdminService.BlockingInterface	getRsAdmin(ServerName sn) n * @return Admin interface for the remote regionserver named sn n * @throws RetriesExhaustedException wrapping a ConnectException if failed
private String	getStrForMax(int max)
String	getVersion(ServerName serverName) May return "0.0.0" when server is not online
int	getVersionNumber(ServerName serverName) May return 0 when server is not online.
boolean	isClusterShutdown()
boolean	isRegionInServerManagerStates(RegionInfo hri)
boolean	isServerDead(ServerName serverName) Check if a server is known to be dead.
ServerManager.ServerLiveState	isServerKnownAndOnline(ServerName serverName) Returns whether the server is online, dead, or unknown.
boolean	isServerOnline(ServerName serverName)
boolean	isServerUnknown(ServerName serverName) Check if a server is unknown.
(package private) void	letRegionServersShutdown()
void	moveFromOnlineToDeadServers(ServerName sn) Called when server has expired.
private HBaseRpcController	newRpcController()
(package private) void	recordNewServerWithLock(ServerName serverName, ServerMetrics sl) Adds the onlineServers list.
void	regionServerReport(ServerName sn, ServerMetrics sl)
(package private) ServerName	regionServerStartup(org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos.RegionServerStartupRequest request, int versionNumber, String version, InetAddress ia) Let the server manager know a new regionserver has come online
void	registerListener(ServerListener listener) Add the listener to the notification list.
void	removeRegion(RegionInfo regionInfo) Called by delete table and similar to notify the ServerManager that a region was removed.
void	removeRegions(List<RegionInfo> regions) Called by delete table and similar to notify the ServerManager that a region was removed.
boolean	removeServerFromDrainList(ServerName sn)
void	sendRegionWarmup(ServerName server, RegionInfo region) Sends a WARMUP RPC to the specified server to warmup the specified region.
void	shutdownCluster()
void	stop() Stop the ServerManager.
boolean	unregisterListener(ServerListener listener) Remove the listener from the notification list.
private void	updateLastFlushedSequenceIds(ServerName sn, ServerMetrics hsl) Updates last flushed sequence Ids for the regions on server sn
void	waitForRegionServers(MonitoredTask status) Wait for the region servers to report in.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

WAIT_ON_REGIONSERVERS_MAXTOSTART

public static final String WAIT_ON_REGIONSERVERS_MAXTOSTART

See Also:

Constant Field Values

WAIT_ON_REGIONSERVERS_MINTOSTART

public static final String WAIT_ON_REGIONSERVERS_MINTOSTART

See Also:

Constant Field Values

WAIT_ON_REGIONSERVERS_TIMEOUT

public static final String WAIT_ON_REGIONSERVERS_TIMEOUT

See Also:

Constant Field Values

WAIT_ON_REGIONSERVERS_INTERVAL

public static final String WAIT_ON_REGIONSERVERS_INTERVAL

See Also:

Constant Field Values

LOG

private static final org.slf4j.Logger LOG

clusterShutdown

private AtomicBoolean clusterShutdown

flushedSequenceIdByRegion

private final ConcurrentNavigableMap<byte[],Long> flushedSequenceIdByRegion

The last flushed sequence id for a region.

storeFlushedSequenceIdsByRegion

private final ConcurrentNavigableMap<byte[],ConcurrentNavigableMap<byte[],Long>> storeFlushedSequenceIdsByRegion

The last flushed sequence id for a store in a region.

onlineServers

private final ConcurrentNavigableMap<ServerName,ServerMetrics> onlineServers

Map of registered servers to their current load

rsAdmins

private final Map<ServerName,org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos.AdminService.BlockingInterface> rsAdmins

Map of admin interfaces per registered regionserver; these interfaces we use to control regionservers out on the cluster

drainingServers

private final ArrayList<ServerName> drainingServers

List of region servers that should not get any more new regions.

master

private final MasterServices master

connection

private final ClusterConnection connection

storage

private final RegionServerList storage

deadservers

private final DeadServer deadservers

maxSkew

private final long maxSkew

warningSkew

private final long warningSkew

rpcControllerFactory

private final RpcControllerFactory rpcControllerFactory

listeners

private List<ServerListener> listeners

Listeners that are called on server events.

Constructor Detail

ServerManager

public ServerManager(MasterServices master,
                     RegionServerList storage)

Constructor.

Method Detail

registerListener

public void registerListener(ServerListener listener)

Add the listener to the notification list.

Parameters:

listener - The ServerListener to register

unregisterListener

public boolean unregisterListener(ServerListener listener)

Remove the listener from the notification list.

Parameters:

listener - The ServerListener to unregister

regionServerStartup

ServerName regionServerStartup(org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos.RegionServerStartupRequest request,
                               int versionNumber,
                               String version,
                               InetAddress ia)
                        throws IOException

Let the server manager know a new regionserver has come online

Parameters:

request - the startup request

versionNumber - the version number of the new regionserver

version - the version of the new regionserver, could contain strings like "SNAPSHOT"

ia - the InetAddress from which request is received

Returns:

The ServerName we know this server as.

Throws:

IOException

updateLastFlushedSequenceIds

private void updateLastFlushedSequenceIds(ServerName sn,
                                          ServerMetrics hsl)

Updates last flushed sequence Ids for the regions on server sn

regionServerReport

public void regionServerReport(ServerName sn,
                               ServerMetrics sl)
                        throws YouAreDeadException

Throws:

YouAreDeadException

checkAndRecordNewServer

boolean checkAndRecordNewServer(ServerName serverName,
                                ServerMetrics sl)

Check is a server of same host and port already exists, if not, or the existed one got a smaller start code, record it.

Parameters:

serverName - the server to check and record

sl - the server load on the server

Returns:

true if the server is recorded, otherwise, false

findDeadServersAndProcess

void findDeadServersAndProcess(Set<ServerName> deadServersFromPE,
                               Set<ServerName> liveServersFromWALDir)

Find out the region servers crashed between the crash of the previous master instance and the current master instance and schedule SCP for them.

Since the RegionServerTracker has already helped us to construct the online servers set by scanning zookeeper, now we can compare the online servers with liveServersFromWALDir to find out whether there are servers which are already dead.

Must be called inside the initialization method of RegionServerTracker to avoid concurrency issue.

Parameters:

deadServersFromPE - the region servers which already have a SCP associated.

liveServersFromWALDir - the live region servers from wal directory.

checkClockSkew

private void checkClockSkew(ServerName serverName,
                            long serverCurrentTime)
                     throws ClockOutOfSyncException

Checks if the clock skew between the server and the master. If the clock skew exceeds the configured max, it will throw an exception; if it exceeds the configured warning threshold, it will log a warning but start normally.

Parameters:

serverName - Incoming servers's name n * @throws ClockOutOfSyncException if the skew exceeds the configured max value

Throws:

ClockOutOfSyncException

checkIsDead

private void checkIsDead(ServerName serverName,
                         String what)
                  throws YouAreDeadException

Called when RegionServer first reports in for duty and thereafter each time it heartbeats to make sure it is has not been figured for dead. If this server is on the dead list, reject it with a YouAreDeadException. If it was dead but came back with a new start code, remove the old entry from the dead list.

Parameters:

what - START or REPORT

Throws:

YouAreDeadException

findServerWithSameHostnamePortWithLock

private ServerName findServerWithSameHostnamePortWithLock(ServerName serverName)

Assumes onlineServers is locked.

Returns:

ServerName with matching hostname and port.

recordNewServerWithLock

void recordNewServerWithLock(ServerName serverName,
                             ServerMetrics sl)

Adds the onlineServers list. onlineServers should be locked.

Parameters:

serverName - The remote servers name.

getLastFlushedSequenceId

public org.apache.hadoop.hbase.shaded.protobuf.generated.ClusterStatusProtos.RegionStoreSequenceIds getLastFlushedSequenceId(byte[] encodedRegionName)

getLoad

public ServerMetrics getLoad(ServerName serverName)

n * @return ServerMetrics if serverName is known else null

getAverageLoad

public double getAverageLoad()

Compute the average load across all region servers. Currently, this uses a very naive computation - just uses the number of regions being served, ignoring stats about number of requests.

Returns:

the average load

countOfRegionServers

public int countOfRegionServers()

Returns the count of active regionservers

getOnlineServers

public Map<ServerName,ServerMetrics> getOnlineServers()

Returns Read-only map of servers to serverinfo

getDeadServers

public DeadServer getDeadServers()

areDeadServersInProgress

public boolean areDeadServersInProgress()

Checks if any dead servers are currently in progress.

Returns:

true if any RS are being processed as dead, false if not

letRegionServersShutdown

void letRegionServersShutdown()

getRegionServersInZK

private List<String> getRegionServersInZK(ZKWatcher zkw)
                                   throws org.apache.zookeeper.KeeperException

Throws:

org.apache.zookeeper.KeeperException

expireServer

public long expireServer(ServerName serverName)

Expire the passed server. Add it to list of dead servers and queue a shutdown processing.

Returns:

pid if we queued a ServerCrashProcedure else Procedure.NO_PROC_ID if we did not (could happen for many reasons including the fact that its this server that is going down or we already have queued an SCP for this server or SCP processing is currently disabled because we are in startup phase).

expireServer

long expireServer(ServerName serverName,
                  boolean force)

moveFromOnlineToDeadServers

public void moveFromOnlineToDeadServers(ServerName sn)

Called when server has expired.

removeServerFromDrainList

public boolean removeServerFromDrainList(ServerName sn)

addServerToDrainList

public boolean addServerToDrainList(ServerName sn)

Add the server to the drain list. n * @return True if the server is added or the server is already on the drain list.

newRpcController

private HBaseRpcController newRpcController()

sendRegionWarmup

public void sendRegionWarmup(ServerName server,
                             RegionInfo region)

Sends a WARMUP RPC to the specified server to warmup the specified region.

A region server could reject the close request because it either does not have the specified region or the region is being split.

Parameters:

server - server to warmup a region

region - region to warmup

closeRegionSilentlyAndWait

public static void closeRegionSilentlyAndWait(ClusterConnection connection,
                                              ServerName server,
                                              RegionInfo region,
                                              long timeout)
                                       throws IOException,
                                              InterruptedException

Contacts a region server and waits up to timeout ms to close the region. This bypasses the active hmaster. Pass -1 as timeout if you do not want to wait on result.

Throws:

IOException

InterruptedException

getRsAdmin

public org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos.AdminService.BlockingInterface getRsAdmin(ServerName sn)
                                                                                                        throws IOException

n * @return Admin interface for the remote regionserver named sn n * @throws RetriesExhaustedException wrapping a ConnectException if failed

Throws:

IOException

getMinToStart

private int getMinToStart()

Calculate min necessary to start. This is not an absolute. It is just a friction that will cause us hang around a bit longer waiting on RegionServers to check-in.

waitForRegionServers

public void waitForRegionServers(MonitoredTask status)
                          throws InterruptedException

Wait for the region servers to report in. We will wait until one of this condition is met: - the master is stopped - the 'hbase.master.wait.on.regionservers.maxtostart' number of region servers is reached - the 'hbase.master.wait.on.regionservers.mintostart' is reached AND there have been no new region server in for 'hbase.master.wait.on.regionservers.interval' time AND the 'hbase.master.wait.on.regionservers.timeout' is reached n

Throws:

InterruptedException

getStrForMax

private String getStrForMax(int max)

getOnlineServersList

public List<ServerName> getOnlineServersList()

Returns A copy of the internal list of online servers.

getOnlineServersListWithPredicator

public List<ServerName> getOnlineServersListWithPredicator(List<ServerName> keys,
                                                           Predicate<ServerMetrics> idleServerPredicator)

Parameters:

keys - The target server name

idleServerPredicator - Evaluates the server on the given load

Returns:

A copy of the internal list of online servers matched by the predicator

getDrainingServersList

public List<ServerName> getDrainingServersList()

Returns A copy of the internal list of draining servers.

isServerOnline

public boolean isServerOnline(ServerName serverName)

isServerKnownAndOnline

public ServerManager.ServerLiveState isServerKnownAndOnline(ServerName serverName)

Returns whether the server is online, dead, or unknown.

isServerDead

public boolean isServerDead(ServerName serverName)

Check if a server is known to be dead. A server can be online, or known to be dead, or unknown to this manager (i.e, not online, not known to be dead either; it is simply not tracked by the master any more, for example, a very old previous instance).

isServerUnknown

public boolean isServerUnknown(ServerName serverName)

Check if a server is unknown. A server can be online, or known to be dead, or unknown to this manager (i.e, not online, not known to be dead either; it is simply not tracked by the master any more, for example, a very old previous instance).

shutdownCluster

public void shutdownCluster()

isClusterShutdown

public boolean isClusterShutdown()

stop

public void stop()

Stop the ServerManager.

createDestinationServersList

public List<ServerName> createDestinationServersList(List<ServerName> serversToExclude)

Creates a list of possible destinations for a region. It contains the online servers, but not the draining or dying servers.

Parameters:

serversToExclude - can be null if there is no server to exclude

createDestinationServersList

public List<ServerName> createDestinationServersList()

Calls createDestinationServersList(java.util.List<org.apache.hadoop.hbase.ServerName>) without server to exclude.

clearDeadServersWithSameHostNameAndPortOfOnlineServer

void clearDeadServersWithSameHostNameAndPortOfOnlineServer()

To clear any dead server with same host name and port of any online server

removeRegion

public void removeRegion(RegionInfo regionInfo)

Called by delete table and similar to notify the ServerManager that a region was removed.

isRegionInServerManagerStates

public boolean isRegionInServerManagerStates(RegionInfo hri)

removeRegions

public void removeRegions(List<RegionInfo> regions)

Called by delete table and similar to notify the ServerManager that a region was removed.

getVersionNumber

public int getVersionNumber(ServerName serverName)

May return 0 when server is not online.

getVersion

public String getVersion(ServerName serverName)

May return "0.0.0" when server is not online

getInfoPort

public int getInfoPort(ServerName serverName)