org.apache.hadoop.hbase

Class HBaseCluster

java.lang.Object

org.apache.hadoop.hbase.HBaseCluster

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.conf.Configurable

Direct Known Subclasses:

DistributedHBaseCluster, MiniHBaseCluster

@InterfaceAudience.Public
public abstract class HBaseCluster
extends Object
implements Closeable, org.apache.hadoop.conf.Configurable

This class defines methods that can help with managing HBase clusters from unit tests and system tests. There are 3 types of cluster deployments:

• MiniHBaseCluster: each server is run in the same JVM in separate threads, used by unit tests
• DistributedHBaseCluster: the cluster is pre-deployed, system and integration tests can interact with the cluster.
• ProcessBasedLocalHBaseCluster: each server is deployed locally but in separate JVMs.

HBaseCluster unifies the way tests interact with the cluster, so that the same test can be run against a mini-cluster during unit test execution, or a distributed cluster having tens/hundreds of nodes during execution of integration tests.

HBaseCluster exposes client-side public interfaces to tests, so that tests does not assume running in a particular mode. Not all the tests are suitable to be run on an actual cluster, and some tests will still need to mock stuff and introspect internal state. For those use cases from unit tests, or if more control is needed, you can use the subclasses directly. In that sense, this class does not abstract away every interface that MiniHBaseCluster or DistributedHBaseCluster provide.

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.hadoop.conf.Configuration	conf
protected org.apache.hadoop.hbase.ClusterMetrics	initialClusterStatus the status of the cluster before we begin
(package private) static org.slf4j.Logger	LOG

Constructor Summary

Constructors

Constructor and Description
HBaseCluster(org.apache.hadoop.conf.Configuration conf) Construct an HBaseCluster

Method Summary

Modifier and Type	Method and Description
abstract void	close() Closes all the resources held open for this cluster.
abstract org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos.AdminService.BlockingInterface	getAdminProtocol(org.apache.hadoop.hbase.ServerName serverName) Returns an AdminProtocol interface to the regionserver
abstract org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos.ClientService.BlockingInterface	getClientProtocol(org.apache.hadoop.hbase.ServerName serverName) Returns a ClientProtocol interface to the regionserver
abstract org.apache.hadoop.hbase.ClusterMetrics	getClusterMetrics() Returns a ClusterMetrics for this HBase cluster.
org.apache.hadoop.conf.Configuration	getConf()
org.apache.hadoop.hbase.ClusterMetrics	getInitialClusterMetrics() Returns a ClusterStatus for this HBase cluster as observed at the starting of the HBaseCluster
abstract org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.MasterService.BlockingInterface	getMasterAdminService() Returns an MasterService.BlockingInterface to the active master
org.apache.hadoop.hbase.ServerName	getServerHoldingMeta() Get the ServerName of region server serving the first hbase:meta region
abstract org.apache.hadoop.hbase.ServerName	getServerHoldingRegion(org.apache.hadoop.hbase.TableName tn, byte[] regionName) Get the ServerName of region server serving the specified region
boolean	isDistributedCluster()
abstract boolean	isKilledRS(org.apache.hadoop.hbase.ServerName serverName) Keeping track of killed servers and being able to check if a particular server was killed makes it possible to do fault tolerance testing for dead servers in a deterministic way.
abstract void	killDataNode(org.apache.hadoop.hbase.ServerName serverName) Kills the datanode process if this is a distributed cluster, otherwise, this causes master to exit doing basic clean up only.
abstract void	killMaster(org.apache.hadoop.hbase.ServerName serverName) Kills the master process if this is a distributed cluster, otherwise, this causes master to exit doing basic clean up only.
abstract void	killNameNode(org.apache.hadoop.hbase.ServerName serverName) Kills the namenode process if this is a distributed cluster, otherwise, this causes master to exit doing basic clean up only.
abstract void	killRegionServer(org.apache.hadoop.hbase.ServerName serverName) Kills the region server process if this is a distributed cluster, otherwise this causes the region server to exit doing basic clean up only.
abstract void	killZkNode(org.apache.hadoop.hbase.ServerName serverName) Kills the zookeeper node process if this is a distributed cluster, otherwise, this causes master to exit doing basic clean up only.
boolean	restoreClusterMetrics(org.apache.hadoop.hbase.ClusterMetrics desiredStatus) Restores the cluster to given state if this is a real cluster, otherwise does nothing.
boolean	restoreInitialStatus() Restores the cluster to it's initial state if this is a real cluster, otherwise does nothing.
abstract void	resumeRegionServer(org.apache.hadoop.hbase.ServerName serverName) Resume the region server
void	setConf(org.apache.hadoop.conf.Configuration conf)
abstract void	shutdown() Shut down the HBase cluster
abstract void	startDataNode(org.apache.hadoop.hbase.ServerName serverName) Starts a new datanode on the given hostname or if this is a mini/local cluster, silently logs warning message.
abstract void	startMaster(String hostname, int port) Starts a new master on the given hostname or if this is a mini/local cluster, starts a master locally.
abstract void	startNameNode(org.apache.hadoop.hbase.ServerName serverName) Starts a new namenode on the given hostname or if this is a mini/local cluster, silently logs warning message.
abstract void	startRegionServer(String hostname, int port) Starts a new region server on the given hostname or if this is a mini/local cluster, starts a region server locally.
abstract void	startZkNode(String hostname, int port) Starts a new zookeeper node on the given hostname or if this is a mini/local cluster, silently logs warning message.
abstract void	stopDataNode(org.apache.hadoop.hbase.ServerName serverName) Stops the datanode if this is a distributed cluster, otherwise silently logs warning message.
abstract void	stopMaster(org.apache.hadoop.hbase.ServerName serverName) Stops the given master, by attempting a gradual stop.
abstract void	stopNameNode(org.apache.hadoop.hbase.ServerName serverName) Stops the namenode if this is a distributed cluster, otherwise silently logs warning message.
abstract void	stopRegionServer(org.apache.hadoop.hbase.ServerName serverName) Stops the given region server, by attempting a gradual stop.
abstract void	stopZkNode(org.apache.hadoop.hbase.ServerName serverName) Stops the region zookeeper if this is a distributed cluster, otherwise silently logs warning message.
abstract void	suspendRegionServer(org.apache.hadoop.hbase.ServerName serverName) Suspend the region server
boolean	waitForActiveAndReadyMaster() Blocks until there is an active master and that master has completed initialization.
abstract boolean	waitForActiveAndReadyMaster(long timeout) Blocks until there is an active master and that master has completed initialization.
void	waitForDatanodesRegistered(int nbDN)
abstract void	waitForDataNodeToStart(org.apache.hadoop.hbase.ServerName serverName, long timeout) Wait for the specified datanode to join the cluster
abstract void	waitForDataNodeToStop(org.apache.hadoop.hbase.ServerName serverName, long timeout) Wait for the specified datanode to stop the thread / process.
abstract void	waitForMasterToStop(org.apache.hadoop.hbase.ServerName serverName, long timeout) Wait for the specified master to stop the thread / process.
void	waitForNamenodeAvailable() Wait for the namenode.
abstract void	waitForNameNodeToStart(org.apache.hadoop.hbase.ServerName serverName, long timeout) Wait for the specified namenode to join the cluster
abstract void	waitForNameNodeToStop(org.apache.hadoop.hbase.ServerName serverName, long timeout) Wait for the specified namenode to stop
void	waitForRegionServerToStart(String hostname, int port, long timeout) Wait for the specified region server to join the cluster
abstract void	waitForRegionServerToStop(org.apache.hadoop.hbase.ServerName serverName, long timeout) Wait for the specified region server to stop the thread / process.
abstract void	waitForZkNodeToStart(org.apache.hadoop.hbase.ServerName serverName, long timeout) Wait for the specified zookeeper node to join the cluster
abstract void	waitForZkNodeToStop(org.apache.hadoop.hbase.ServerName serverName, long timeout) Wait for the specified zookeeper node to stop the thread / process.
abstract void	waitUntilShutDown() Wait for HBase Cluster to shut down.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

static final org.slf4j.Logger LOG

conf

protected org.apache.hadoop.conf.Configuration conf

initialClusterStatus

protected org.apache.hadoop.hbase.ClusterMetrics initialClusterStatus

the status of the cluster before we begin

Constructor Detail

HBaseCluster

public HBaseCluster(org.apache.hadoop.conf.Configuration conf)

Construct an HBaseCluster

Parameters:

conf - Configuration to be used for cluster

Method Detail

setConf

public void setConf(org.apache.hadoop.conf.Configuration conf)

Specified by:

setConf in interface org.apache.hadoop.conf.Configurable

getConf

public org.apache.hadoop.conf.Configuration getConf()

Specified by:

getConf in interface org.apache.hadoop.conf.Configurable

getClusterMetrics

public abstract org.apache.hadoop.hbase.ClusterMetrics getClusterMetrics()
                                                                  throws IOException

Returns a ClusterMetrics for this HBase cluster.

Throws:

IOException

See Also:

getInitialClusterMetrics()

getInitialClusterMetrics

public org.apache.hadoop.hbase.ClusterMetrics getInitialClusterMetrics()
                                                                throws IOException

Returns a ClusterStatus for this HBase cluster as observed at the starting of the HBaseCluster

Throws:

IOException

getMasterAdminService

public abstract org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.MasterService.BlockingInterface getMasterAdminService()
                                                                                                                              throws IOException

Returns an MasterService.BlockingInterface to the active master

Throws:

IOException

getAdminProtocol

public abstract org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos.AdminService.BlockingInterface getAdminProtocol(org.apache.hadoop.hbase.ServerName serverName)
                                                                                                                       throws IOException

Returns an AdminProtocol interface to the regionserver

Throws:

IOException

getClientProtocol

public abstract org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos.ClientService.BlockingInterface getClientProtocol(org.apache.hadoop.hbase.ServerName serverName)
                                                                                                                          throws IOException

Returns a ClientProtocol interface to the regionserver

Throws:

IOException

startRegionServer

public abstract void startRegionServer(String hostname,
                                       int port)
                                throws IOException

Starts a new region server on the given hostname or if this is a mini/local cluster, starts a region server locally.

Parameters:

hostname - the hostname to start the regionserver on

Throws:

IOException - if something goes wrong

killRegionServer

public abstract void killRegionServer(org.apache.hadoop.hbase.ServerName serverName)
                               throws IOException

Kills the region server process if this is a distributed cluster, otherwise this causes the region server to exit doing basic clean up only.

Throws:

IOException - if something goes wrong

isKilledRS

public abstract boolean isKilledRS(org.apache.hadoop.hbase.ServerName serverName)

Keeping track of killed servers and being able to check if a particular server was killed makes it possible to do fault tolerance testing for dead servers in a deterministic way. A concrete example of such case is - killing servers and waiting for all regions of a particular table to be assigned. We can check for server column in META table and that its value is not one of the killed servers.

stopRegionServer

public abstract void stopRegionServer(org.apache.hadoop.hbase.ServerName serverName)
                               throws IOException

Stops the given region server, by attempting a gradual stop.

Throws:

IOException - if something goes wrong

waitForRegionServerToStart

public void waitForRegionServerToStart(String hostname,
                                       int port,
                                       long timeout)
                                throws IOException

Wait for the specified region server to join the cluster

Throws:

IOException - if something goes wrong or timeout occurs

waitForRegionServerToStop

public abstract void waitForRegionServerToStop(org.apache.hadoop.hbase.ServerName serverName,
                                               long timeout)
                                        throws IOException

Wait for the specified region server to stop the thread / process.

Throws:

IOException - if something goes wrong or timeout occurs

suspendRegionServer

public abstract void suspendRegionServer(org.apache.hadoop.hbase.ServerName serverName)
                                  throws IOException

Suspend the region server

Parameters:

serverName - the hostname to suspend the regionserver on

Throws:

IOException - if something goes wrong

resumeRegionServer

public abstract void resumeRegionServer(org.apache.hadoop.hbase.ServerName serverName)
                                 throws IOException

Resume the region server

Parameters:

serverName - the hostname to resume the regionserver on

Throws:

IOException - if something goes wrong

startZkNode

public abstract void startZkNode(String hostname,
                                 int port)
                          throws IOException

Starts a new zookeeper node on the given hostname or if this is a mini/local cluster, silently logs warning message.

Parameters:

hostname - the hostname to start the regionserver on

Throws:

IOException - if something goes wrong

killZkNode

public abstract void killZkNode(org.apache.hadoop.hbase.ServerName serverName)
                         throws IOException

Kills the zookeeper node process if this is a distributed cluster, otherwise, this causes master to exit doing basic clean up only.

Throws:

IOException - if something goes wrong

stopZkNode

public abstract void stopZkNode(org.apache.hadoop.hbase.ServerName serverName)
                         throws IOException

Stops the region zookeeper if this is a distributed cluster, otherwise silently logs warning message.

Throws:

IOException - if something goes wrong

waitForZkNodeToStart

public abstract void waitForZkNodeToStart(org.apache.hadoop.hbase.ServerName serverName,
                                          long timeout)
                                   throws IOException

Wait for the specified zookeeper node to join the cluster

Throws:

IOException - if something goes wrong or timeout occurs

waitForZkNodeToStop

public abstract void waitForZkNodeToStop(org.apache.hadoop.hbase.ServerName serverName,
                                         long timeout)
                                  throws IOException

Wait for the specified zookeeper node to stop the thread / process.

Throws:

IOException - if something goes wrong or timeout occurs

startDataNode

public abstract void startDataNode(org.apache.hadoop.hbase.ServerName serverName)
                            throws IOException

Starts a new datanode on the given hostname or if this is a mini/local cluster, silently logs warning message.

Throws:

IOException - if something goes wrong

killDataNode

public abstract void killDataNode(org.apache.hadoop.hbase.ServerName serverName)
                           throws IOException

Kills the datanode process if this is a distributed cluster, otherwise, this causes master to exit doing basic clean up only.

Throws:

IOException - if something goes wrong

stopDataNode

public abstract void stopDataNode(org.apache.hadoop.hbase.ServerName serverName)
                           throws IOException

Stops the datanode if this is a distributed cluster, otherwise silently logs warning message.

Throws:

IOException - if something goes wrong

waitForDataNodeToStart

public abstract void waitForDataNodeToStart(org.apache.hadoop.hbase.ServerName serverName,
                                            long timeout)
                                     throws IOException

Wait for the specified datanode to join the cluster

Throws:

IOException - if something goes wrong or timeout occurs

waitForDataNodeToStop

public abstract void waitForDataNodeToStop(org.apache.hadoop.hbase.ServerName serverName,
                                           long timeout)
                                    throws IOException

Wait for the specified datanode to stop the thread / process.

Throws:

IOException - if something goes wrong or timeout occurs

startNameNode

public abstract void startNameNode(org.apache.hadoop.hbase.ServerName serverName)
                            throws IOException

Starts a new namenode on the given hostname or if this is a mini/local cluster, silently logs warning message.

Throws:

IOException - if something goes wrong

killNameNode

public abstract void killNameNode(org.apache.hadoop.hbase.ServerName serverName)
                           throws IOException

Kills the namenode process if this is a distributed cluster, otherwise, this causes master to exit doing basic clean up only.

Throws:

IOException - if something goes wrong

stopNameNode

public abstract void stopNameNode(org.apache.hadoop.hbase.ServerName serverName)
                           throws IOException

Stops the namenode if this is a distributed cluster, otherwise silently logs warning message.

Throws:

IOException - if something goes wrong

waitForNameNodeToStart

public abstract void waitForNameNodeToStart(org.apache.hadoop.hbase.ServerName serverName,
                                            long timeout)
                                     throws IOException

Wait for the specified namenode to join the cluster

Throws:

IOException - if something goes wrong or timeout occurs

waitForNameNodeToStop

public abstract void waitForNameNodeToStop(org.apache.hadoop.hbase.ServerName serverName,
                                           long timeout)
                                    throws IOException

Wait for the specified namenode to stop

Throws:

IOException - if something goes wrong or timeout occurs

startMaster

public abstract void startMaster(String hostname,
                                 int port)
                          throws IOException

Starts a new master on the given hostname or if this is a mini/local cluster, starts a master locally.

Parameters:

hostname - the hostname to start the master on

Throws:

IOException - if something goes wrong

killMaster

public abstract void killMaster(org.apache.hadoop.hbase.ServerName serverName)
                         throws IOException

Kills the master process if this is a distributed cluster, otherwise, this causes master to exit doing basic clean up only.

Throws:

IOException - if something goes wrong

stopMaster

public abstract void stopMaster(org.apache.hadoop.hbase.ServerName serverName)
                         throws IOException

Stops the given master, by attempting a gradual stop.

Throws:

IOException - if something goes wrong

waitForMasterToStop

public abstract void waitForMasterToStop(org.apache.hadoop.hbase.ServerName serverName,
                                         long timeout)
                                  throws IOException

Wait for the specified master to stop the thread / process.

Throws:

IOException - if something goes wrong or timeout occurs

waitForActiveAndReadyMaster

public boolean waitForActiveAndReadyMaster()
                                    throws IOException

Blocks until there is an active master and that master has completed initialization.

Returns:

true if an active master becomes available. false if there are no masters left.

Throws:

IOException - if something goes wrong or timeout occurs

waitForActiveAndReadyMaster

public abstract boolean waitForActiveAndReadyMaster(long timeout)
                                             throws IOException

Blocks until there is an active master and that master has completed initialization.

Parameters:

timeout - the timeout limit in ms

Returns:

true if an active master becomes available. false if there are no masters left.

Throws:

IOException

waitUntilShutDown

public abstract void waitUntilShutDown()
                                throws IOException

Wait for HBase Cluster to shut down.

Throws:

IOException

shutdown

public abstract void shutdown()
                       throws IOException

Shut down the HBase cluster

Throws:

IOException

restoreInitialStatus

public boolean restoreInitialStatus()
                             throws IOException

Restores the cluster to it's initial state if this is a real cluster, otherwise does nothing. This is a best effort restore. If the servers are not reachable, or insufficient permissions, etc. restoration might be partial.

Returns:

whether restoration is complete

Throws:

IOException

restoreClusterMetrics

public boolean restoreClusterMetrics(org.apache.hadoop.hbase.ClusterMetrics desiredStatus)
                              throws IOException

Restores the cluster to given state if this is a real cluster, otherwise does nothing. This is a best effort restore. If the servers are not reachable, or insufficient permissions, etc. restoration might be partial.

Returns:

whether restoration is complete

Throws:

IOException

getServerHoldingMeta

public org.apache.hadoop.hbase.ServerName getServerHoldingMeta()
                                                        throws IOException

Get the ServerName of region server serving the first hbase:meta region

Throws:

IOException

getServerHoldingRegion

public abstract org.apache.hadoop.hbase.ServerName getServerHoldingRegion(org.apache.hadoop.hbase.TableName tn,
                                                                          byte[] regionName)
                                                                   throws IOException

Get the ServerName of region server serving the specified region

Parameters:

regionName - Name of the region in bytes

tn - Table name that has the region.

Returns:

ServerName that hosts the region or null

Throws:

IOException

isDistributedCluster

public boolean isDistributedCluster()

Returns:

whether we are interacting with a distributed cluster as opposed to an in-process mini/local cluster.

close

public abstract void close()
                    throws IOException

Closes all the resources held open for this cluster. Note that this call does not shutdown the cluster.

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException

See Also:

shutdown()

waitForNamenodeAvailable

public void waitForNamenodeAvailable()
                              throws InterruptedException

Wait for the namenode. n

Throws:

InterruptedException

waitForDatanodesRegistered

public void waitForDatanodesRegistered(int nbDN)
                                throws Exception

Throws:

Exception