org.apache.hadoop.hbase

Class DistributedHBaseCluster

java.lang.Object

org.apache.hadoop.hbase.HBaseCluster

org.apache.hadoop.hbase.DistributedHBaseCluster

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.conf.Configurable

@InterfaceAudience.Private
public class DistributedHBaseCluster
extends HBaseCluster

Manages the interactions with an already deployed distributed cluster (as opposed to a pseudo-distributed, or mini/local cluster). This is used by integration and system tests.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
private static class	DistributedHBaseCluster.ServerNameIgnoreStartCodeComparator

Field Summary

Fields

Modifier and Type	Field and Description
private org.apache.hadoop.hbase.client.Admin	admin
private ClusterManager	clusterManager
private org.apache.hadoop.hbase.client.Connection	connection
private Set<org.apache.hadoop.hbase.ServerName>	killedRegionServers List of RegionServers killed so far.

Fields inherited from class org.apache.hadoop.hbase.HBaseCluster

conf, initialClusterStatus, LOG

Constructor Summary

Constructors

Constructor and Description
DistributedHBaseCluster(org.apache.hadoop.conf.Configuration conf, ClusterManager clusterManager)

Method Summary

Modifier and Type	Method and Description
void	close() Closes all the resources held open for this cluster.
org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos.AdminService.BlockingInterface	getAdminProtocol(org.apache.hadoop.hbase.ServerName serverName) Returns an AdminProtocol interface to the regionserver
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos.ClientService.BlockingInterface	getClientProtocol(org.apache.hadoop.hbase.ServerName serverName) Returns a ClientProtocol interface to the regionserver
ClusterManager	getClusterManager()
org.apache.hadoop.hbase.ClusterMetrics	getClusterMetrics() Returns a ClusterStatus for this HBase cluster
org.apache.hadoop.hbase.ClusterMetrics	getInitialClusterMetrics() Returns a ClusterStatus for this HBase cluster as observed at the starting of the HBaseCluster
org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.MasterService.BlockingInterface	getMasterAdminService() Returns an MasterService.BlockingInterface to the active master
org.apache.hadoop.hbase.ServerName	getServerHoldingRegion(org.apache.hadoop.hbase.TableName tn, byte[] regionName) Get the ServerName of region server serving the specified region
boolean	isDistributedCluster()
boolean	isKilledRS(org.apache.hadoop.hbase.ServerName serverName) Keeping track of killed servers and being able to check if a particular server was killed makes it possible to do fault tolerance testing for dead servers in a deterministic way.
void	killDataNode(org.apache.hadoop.hbase.ServerName serverName) Kills the datanode process if this is a distributed cluster, otherwise, this causes master to exit doing basic clean up only.
void	killMaster(org.apache.hadoop.hbase.ServerName serverName) Kills the master process if this is a distributed cluster, otherwise, this causes master to exit doing basic clean up only.
void	killRegionServer(org.apache.hadoop.hbase.ServerName serverName) Kills the region server process if this is a distributed cluster, otherwise this causes the region server to exit doing basic clean up only.
void	killZkNode(org.apache.hadoop.hbase.ServerName serverName) Kills the zookeeper node process if this is a distributed cluster, otherwise, this causes master to exit doing basic clean up only.
protected boolean	restoreAdmin()
boolean	restoreClusterMetrics(org.apache.hadoop.hbase.ClusterMetrics initial) Restores the cluster to given state if this is a real cluster, otherwise does nothing.
protected boolean	restoreMasters(org.apache.hadoop.hbase.ClusterMetrics initial, org.apache.hadoop.hbase.ClusterMetrics current)
protected boolean	restoreRegionServers(org.apache.hadoop.hbase.ClusterMetrics initial, org.apache.hadoop.hbase.ClusterMetrics current)
void	setClusterManager(ClusterManager clusterManager)
void	shutdown() Shut down the HBase cluster
void	startDataNode(org.apache.hadoop.hbase.ServerName serverName) Starts a new datanode on the given hostname or if this is a mini/local cluster, silently logs warning message.
void	startMaster(String hostname, int port) Starts a new master on the given hostname or if this is a mini/local cluster, starts a master locally.
void	startRegionServer(String hostname, int port) Starts a new region server on the given hostname or if this is a mini/local cluster, starts a region server locally.
void	startZkNode(String hostname, int port) Starts a new zookeeper node on the given hostname or if this is a mini/local cluster, silently logs warning message.
void	stopDataNode(org.apache.hadoop.hbase.ServerName serverName) Stops the datanode if this is a distributed cluster, otherwise silently logs warning message.
void	stopMaster(org.apache.hadoop.hbase.ServerName serverName) Stops the given master, by attempting a gradual stop.
void	stopRegionServer(org.apache.hadoop.hbase.ServerName serverName) Stops the given region server, by attempting a gradual stop.
void	stopZkNode(org.apache.hadoop.hbase.ServerName serverName) Stops the region zookeeper if this is a distributed cluster, otherwise silently logs warning message.
boolean	waitForActiveAndReadyMaster(long timeout) Blocks until there is an active master and that master has completed initialization.
void	waitForDataNodeToStart(org.apache.hadoop.hbase.ServerName serverName, long timeout) Wait for the specified datanode to join the cluster
void	waitForDataNodeToStop(org.apache.hadoop.hbase.ServerName serverName, long timeout) Wait for the specified datanode to stop the thread / process.
void	waitForMasterToStop(org.apache.hadoop.hbase.ServerName serverName, long timeout) Wait for the specified master to stop the thread / process.
void	waitForRegionServerToStop(org.apache.hadoop.hbase.ServerName serverName, long timeout) Wait for the specified region server to stop the thread / process.
private void	waitForServiceToStart(ClusterManager.ServiceType service, org.apache.hadoop.hbase.ServerName serverName, long timeout)
private void	waitForServiceToStop(ClusterManager.ServiceType service, org.apache.hadoop.hbase.ServerName serverName, long timeout)
void	waitForZkNodeToStart(org.apache.hadoop.hbase.ServerName serverName, long timeout) Wait for the specified zookeeper node to join the cluster
void	waitForZkNodeToStop(org.apache.hadoop.hbase.ServerName serverName, long timeout) Wait for the specified zookeeper node to stop the thread / process.
void	waitUntilShutDown() Wait for HBase Cluster to shut down.

Methods inherited from class org.apache.hadoop.hbase.HBaseCluster

getConf, getServerHoldingMeta, restoreInitialStatus, setConf, waitForActiveAndReadyMaster, waitForDatanodesRegistered, waitForNamenodeAvailable, waitForRegionServerToStart

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

admin

private org.apache.hadoop.hbase.client.Admin admin

connection

private final org.apache.hadoop.hbase.client.Connection connection

clusterManager

private ClusterManager clusterManager

killedRegionServers

private Set<org.apache.hadoop.hbase.ServerName> killedRegionServers

List of RegionServers killed so far. ServerName also comprises startCode of a server, so any restarted instances of the same server will have different ServerName and will not coincide with past dead ones. So there's no need to cleanup this list.

Constructor Detail

DistributedHBaseCluster

public DistributedHBaseCluster(org.apache.hadoop.conf.Configuration conf,
                               ClusterManager clusterManager)
                        throws IOException

Throws:

IOException

Method Detail

setClusterManager

public void setClusterManager(ClusterManager clusterManager)

getClusterManager

public ClusterManager getClusterManager()

getClusterMetrics

public org.apache.hadoop.hbase.ClusterMetrics getClusterMetrics()
                                                         throws IOException

Returns a ClusterStatus for this HBase cluster

Specified by:

getClusterMetrics in class HBaseCluster

Throws:

IOException

See Also:

HBaseCluster.getInitialClusterMetrics()

getInitialClusterMetrics

public org.apache.hadoop.hbase.ClusterMetrics getInitialClusterMetrics()
                                                                throws IOException

Description copied from class: HBaseCluster

Returns a ClusterStatus for this HBase cluster as observed at the starting of the HBaseCluster

Overrides:

getInitialClusterMetrics in class HBaseCluster

Throws:

IOException

close

public void close()
           throws IOException

Description copied from class: HBaseCluster

Closes all the resources held open for this cluster. Note that this call does not shutdown the cluster.

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Specified by:

close in class HBaseCluster

Throws:

IOException

See Also:

HBaseCluster.shutdown()

getAdminProtocol

public org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos.AdminService.BlockingInterface getAdminProtocol(org.apache.hadoop.hbase.ServerName serverName)
                                                                                                              throws IOException

Description copied from class: HBaseCluster

Returns an AdminProtocol interface to the regionserver

Specified by:

getAdminProtocol in class HBaseCluster

Throws:

IOException

getClientProtocol

public org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos.ClientService.BlockingInterface getClientProtocol(org.apache.hadoop.hbase.ServerName serverName)
                                                                                                                 throws IOException

Description copied from class: HBaseCluster

Returns a ClientProtocol interface to the regionserver

Specified by:

getClientProtocol in class HBaseCluster

Throws:

IOException

startRegionServer

public void startRegionServer(String hostname,
                              int port)
                       throws IOException

Description copied from class: HBaseCluster

Starts a new region server on the given hostname or if this is a mini/local cluster, starts a region server locally.

Specified by:

startRegionServer in class HBaseCluster

Parameters:

hostname - the hostname to start the regionserver on

Throws:

IOException - if something goes wrong

killRegionServer

public void killRegionServer(org.apache.hadoop.hbase.ServerName serverName)
                      throws IOException

Description copied from class: HBaseCluster

Kills the region server process if this is a distributed cluster, otherwise this causes the region server to exit doing basic clean up only.

Specified by:

killRegionServer in class HBaseCluster

Throws:

IOException - if something goes wrong

isKilledRS

public boolean isKilledRS(org.apache.hadoop.hbase.ServerName serverName)

Description copied from class: HBaseCluster

Keeping track of killed servers and being able to check if a particular server was killed makes it possible to do fault tolerance testing for dead servers in a deterministic way. A concrete example of such case is - killing servers and waiting for all regions of a particular table to be assigned. We can check for server column in META table and that its value is not one of the killed servers.

Specified by:

isKilledRS in class HBaseCluster

stopRegionServer

public void stopRegionServer(org.apache.hadoop.hbase.ServerName serverName)
                      throws IOException

Description copied from class: HBaseCluster

Stops the given region server, by attempting a gradual stop.

Specified by:

stopRegionServer in class HBaseCluster

Throws:

IOException - if something goes wrong

waitForRegionServerToStop

public void waitForRegionServerToStop(org.apache.hadoop.hbase.ServerName serverName,
                                      long timeout)
                               throws IOException

Description copied from class: HBaseCluster

Wait for the specified region server to stop the thread / process.

Specified by:

waitForRegionServerToStop in class HBaseCluster

Throws:

IOException - if something goes wrong or timeout occurs

startZkNode

public void startZkNode(String hostname,
                        int port)
                 throws IOException

Description copied from class: HBaseCluster

Starts a new zookeeper node on the given hostname or if this is a mini/local cluster, silently logs warning message.

Specified by:

startZkNode in class HBaseCluster

Parameters:

hostname - the hostname to start the regionserver on

Throws:

IOException - if something goes wrong

killZkNode

public void killZkNode(org.apache.hadoop.hbase.ServerName serverName)
                throws IOException

Description copied from class: HBaseCluster

Kills the zookeeper node process if this is a distributed cluster, otherwise, this causes master to exit doing basic clean up only.

Specified by:

killZkNode in class HBaseCluster

Throws:

IOException - if something goes wrong

stopZkNode

public void stopZkNode(org.apache.hadoop.hbase.ServerName serverName)
                throws IOException

Description copied from class: HBaseCluster

Stops the region zookeeper if this is a distributed cluster, otherwise silently logs warning message.

Specified by:

stopZkNode in class HBaseCluster

Throws:

IOException - if something goes wrong

waitForZkNodeToStart

public void waitForZkNodeToStart(org.apache.hadoop.hbase.ServerName serverName,
                                 long timeout)
                          throws IOException

Description copied from class: HBaseCluster

Wait for the specified zookeeper node to join the cluster

Specified by:

waitForZkNodeToStart in class HBaseCluster

Throws:

IOException - if something goes wrong or timeout occurs

waitForZkNodeToStop

public void waitForZkNodeToStop(org.apache.hadoop.hbase.ServerName serverName,
                                long timeout)
                         throws IOException

Description copied from class: HBaseCluster

Wait for the specified zookeeper node to stop the thread / process.

Specified by:

waitForZkNodeToStop in class HBaseCluster

Throws:

IOException - if something goes wrong or timeout occurs

startDataNode

public void startDataNode(org.apache.hadoop.hbase.ServerName serverName)
                   throws IOException

Description copied from class: HBaseCluster

Starts a new datanode on the given hostname or if this is a mini/local cluster, silently logs warning message.

Specified by:

startDataNode in class HBaseCluster

Throws:

IOException - if something goes wrong

killDataNode

public void killDataNode(org.apache.hadoop.hbase.ServerName serverName)
                  throws IOException

Description copied from class: HBaseCluster

Kills the datanode process if this is a distributed cluster, otherwise, this causes master to exit doing basic clean up only.

Specified by:

killDataNode in class HBaseCluster

Throws:

IOException - if something goes wrong

stopDataNode

public void stopDataNode(org.apache.hadoop.hbase.ServerName serverName)
                  throws IOException

Description copied from class: HBaseCluster

Stops the datanode if this is a distributed cluster, otherwise silently logs warning message.

Specified by:

stopDataNode in class HBaseCluster

Throws:

IOException - if something goes wrong

waitForDataNodeToStart

public void waitForDataNodeToStart(org.apache.hadoop.hbase.ServerName serverName,
                                   long timeout)
                            throws IOException

Description copied from class: HBaseCluster

Wait for the specified datanode to join the cluster

Specified by:

waitForDataNodeToStart in class HBaseCluster

Throws:

IOException - if something goes wrong or timeout occurs

waitForDataNodeToStop

public void waitForDataNodeToStop(org.apache.hadoop.hbase.ServerName serverName,
                                  long timeout)
                           throws IOException

Description copied from class: HBaseCluster

Wait for the specified datanode to stop the thread / process.

Specified by:

waitForDataNodeToStop in class HBaseCluster

Throws:

IOException - if something goes wrong or timeout occurs

waitForServiceToStop

private void waitForServiceToStop(ClusterManager.ServiceType service,
                                  org.apache.hadoop.hbase.ServerName serverName,
                                  long timeout)
                           throws IOException

Throws:

IOException

waitForServiceToStart

private void waitForServiceToStart(ClusterManager.ServiceType service,
                                   org.apache.hadoop.hbase.ServerName serverName,
                                   long timeout)
                            throws IOException

Throws:

IOException

getMasterAdminService

public org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.MasterService.BlockingInterface getMasterAdminService()
                                                                                                                     throws IOException

Description copied from class: HBaseCluster

Returns an MasterService.BlockingInterface to the active master

Specified by:

getMasterAdminService in class HBaseCluster

Throws:

IOException

startMaster

public void startMaster(String hostname,
                        int port)
                 throws IOException

Description copied from class: HBaseCluster

Starts a new master on the given hostname or if this is a mini/local cluster, starts a master locally.

Specified by:

startMaster in class HBaseCluster

Parameters:

hostname - the hostname to start the master on

Throws:

IOException - if something goes wrong

killMaster

public void killMaster(org.apache.hadoop.hbase.ServerName serverName)
                throws IOException

Description copied from class: HBaseCluster

Kills the master process if this is a distributed cluster, otherwise, this causes master to exit doing basic clean up only.

Specified by:

killMaster in class HBaseCluster

Throws:

IOException - if something goes wrong

stopMaster

public void stopMaster(org.apache.hadoop.hbase.ServerName serverName)
                throws IOException

Description copied from class: HBaseCluster

Stops the given master, by attempting a gradual stop.

Specified by:

stopMaster in class HBaseCluster

Throws:

IOException - if something goes wrong

waitForMasterToStop

public void waitForMasterToStop(org.apache.hadoop.hbase.ServerName serverName,
                                long timeout)
                         throws IOException

Description copied from class: HBaseCluster

Wait for the specified master to stop the thread / process.

Specified by:

waitForMasterToStop in class HBaseCluster

Throws:

IOException - if something goes wrong or timeout occurs

waitForActiveAndReadyMaster

public boolean waitForActiveAndReadyMaster(long timeout)
                                    throws IOException

Description copied from class: HBaseCluster

Blocks until there is an active master and that master has completed initialization.

Specified by:

waitForActiveAndReadyMaster in class HBaseCluster

Parameters:

timeout - the timeout limit in ms

Returns:

true if an active master becomes available. false if there are no masters left.

Throws:

IOException

getServerHoldingRegion

public org.apache.hadoop.hbase.ServerName getServerHoldingRegion(org.apache.hadoop.hbase.TableName tn,
                                                                 byte[] regionName)
                                                          throws IOException

Description copied from class: HBaseCluster

Get the ServerName of region server serving the specified region

Specified by:

getServerHoldingRegion in class HBaseCluster

Parameters:

tn - Table name that has the region.

regionName - Name of the region in bytes

Returns:

ServerName that hosts the region or null

Throws:

IOException

waitUntilShutDown

public void waitUntilShutDown()

Description copied from class: HBaseCluster

Wait for HBase Cluster to shut down.

Specified by:

waitUntilShutDown in class HBaseCluster

shutdown

public void shutdown()
              throws IOException

Description copied from class: HBaseCluster

Shut down the HBase cluster

Specified by:

shutdown in class HBaseCluster

Throws:

IOException

isDistributedCluster

public boolean isDistributedCluster()

Overrides:

isDistributedCluster in class HBaseCluster

Returns:

whether we are interacting with a distributed cluster as opposed to an in-process mini/local cluster.

restoreClusterMetrics

public boolean restoreClusterMetrics(org.apache.hadoop.hbase.ClusterMetrics initial)
                              throws IOException

Description copied from class: HBaseCluster

Restores the cluster to given state if this is a real cluster, otherwise does nothing. This is a best effort restore. If the servers are not reachable, or insufficient permissions, etc. restoration might be partial.

Overrides:

restoreClusterMetrics in class HBaseCluster

Returns:

whether restoration is complete

Throws:

IOException

restoreMasters

protected boolean restoreMasters(org.apache.hadoop.hbase.ClusterMetrics initial,
                                 org.apache.hadoop.hbase.ClusterMetrics current)

restoreRegionServers

protected boolean restoreRegionServers(org.apache.hadoop.hbase.ClusterMetrics initial,
                                       org.apache.hadoop.hbase.ClusterMetrics current)

restoreAdmin

protected boolean restoreAdmin()
                        throws IOException

Throws:

IOException