org.apache.hadoop.hbase.replication

Class ReplicationQueuesZKImpl

java.lang.Object

org.apache.hadoop.hbase.replication.ReplicationStateZKBase

org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl

All Implemented Interfaces:

ReplicationQueues

@InterfaceAudience.Private
public class ReplicationQueuesZKImpl
extends ReplicationStateZKBase
implements ReplicationQueues

This class provides an implementation of the interface using ZooKeeper. The base znode that this class works at is the myQueuesZnode. The myQueuesZnode contains a list of all outstanding WAL files on this region server that need to be replicated. The myQueuesZnode is the regionserver name (a concatenation of the region server’s hostname, client port and start code). For example: /hbase/replication/rs/hostname.example.org,6020,1234 Within this znode, the region server maintains a set of WAL replication queues. These queues are represented by child znodes named using there give queue id. For example: /hbase/replication/rs/hostname.example.org,6020,1234/1 /hbase/replication/rs/hostname.example.org,6020,1234/2 Each queue has one child znode for every WAL that still needs to be replicated. The value of these WAL child znodes is the latest position that has been replicated. This position is updated every time a WAL entry is replicated. For example: /hbase/replication/rs/hostname.example.org,6020,1234/1/23522342.23422 [VALUE: 254]

Field Summary

Fields

Modifier and Type	Field and Description
private static org.slf4j.Logger	LOG
private String	myQueuesZnode Znode containing all replication queues for this region server.

Fields inherited from class org.apache.hadoop.hbase.replication.ReplicationStateZKBase

abortable, conf, DISABLED_ZNODE_BYTES, ENABLED_ZNODE_BYTES, hfileRefsZNode, ourClusterKey, peerStateNodeName, peersZNode, queuesZNode, replicationZNode, tableCFsNodeName, zookeeper, ZOOKEEPER_ZNODE_REPLICATION_HFILE_REFS_DEFAULT, ZOOKEEPER_ZNODE_REPLICATION_HFILE_REFS_KEY

Constructor Summary

Constructors

Constructor and Description
ReplicationQueuesZKImpl(ReplicationQueuesArguments args)
ReplicationQueuesZKImpl(ZKWatcher zk, org.apache.hadoop.conf.Configuration conf, Abortable abortable)

Method Summary

Modifier and Type	Method and Description
void	addHFileRefs(String peerId, List<Pair<org.apache.hadoop.fs.Path,org.apache.hadoop.fs.Path>> pairs) Add new hfile references to the queue.
void	addLog(String queueId, String filename) Add a new WAL file to the given queue.
void	addPeerToHFileRefs(String peerId) Add a peer to hfile reference queue if peer does not exist.
Pair<String,SortedSet<String>>	claimQueue(String regionserver, String queueId) Take ownership for the queue identified by queueId and belongs to a dead region server.
List<String>	getAllQueues() Get a list of all queues for this region server.
List<String>	getListOfReplicators() Get a list of all region servers that have outstanding replication queues.
long	getLogPosition(String queueId, String filename) Get the current position for a specific WAL in a given queue.
List<String>	getLogsInQueue(String queueId) Get a list of all WALs in the given queue.
List<String>	getUnClaimedQueueIds(String regionserver) Get queueIds from a dead region server, whose queues has not been claimed by other region servers.
void	init(String serverName) Initialize the region server replication queue interface.
boolean	isThisOurRegionServer(String regionserver) Checks if the provided znode is the same as this region server's
private Pair<String,SortedSet<String>>	moveQueueUsingMulti(String znode, String peerId) It "atomically" copies one peer's wals queue from another dead region server and returns them all sorted.
void	removeAllQueues() Remove all replication queues for this region server.
void	removeHFileRefs(String peerId, List<String> files) Remove hfile references from the queue.
void	removeLog(String queueId, String filename) Remove an WAL file from the given queue.
void	removePeerFromHFileRefs(String peerId) Remove a peer from hfile reference queue.
void	removeQueue(String queueId) Remove a replication queue.
void	removeReplicatorIfQueueIsEmpty(String regionserver) Remove the znode of region server if the queue is empty.
void	setLogPosition(String queueId, String filename, long position) Set the current position for a specific WAL in a given queue.

Methods inherited from class org.apache.hadoop.hbase.replication.ReplicationStateZKBase

getListOfReplicatorsZK, getPeerNode, getPeerStateNode, getTableCFsNode, isPeerPath, peerExists, toByteArray

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

myQueuesZnode

private String myQueuesZnode

Znode containing all replication queues for this region server.

LOG

private static final org.slf4j.Logger LOG

Constructor Detail

ReplicationQueuesZKImpl

public ReplicationQueuesZKImpl(ReplicationQueuesArguments args)

ReplicationQueuesZKImpl

public ReplicationQueuesZKImpl(ZKWatcher zk,
                               org.apache.hadoop.conf.Configuration conf,
                               Abortable abortable)

Method Detail

init

public void init(String serverName)
          throws ReplicationException

Description copied from interface: ReplicationQueues

Initialize the region server replication queue interface.

Specified by:

init in interface ReplicationQueues

Parameters:

serverName - The server name of the region server that owns the replication queues this interface manages.

Throws:

ReplicationException

getListOfReplicators

public List<String> getListOfReplicators()
                                  throws ReplicationException

Description copied from interface: ReplicationQueues

Get a list of all region servers that have outstanding replication queues. These servers could be alive, dead or from a previous run of the cluster.

Specified by:

getListOfReplicators in interface ReplicationQueues

Returns:

a list of server names

Throws:

ReplicationException

removeQueue

public void removeQueue(String queueId)

Description copied from interface: ReplicationQueues

Remove a replication queue.

Specified by:

removeQueue in interface ReplicationQueues

Parameters:

queueId - a String that identifies the queue.

addLog

public void addLog(String queueId,
                   String filename)
            throws ReplicationException

Description copied from interface: ReplicationQueues

Add a new WAL file to the given queue. If the queue does not exist it is created.

Specified by:

addLog in interface ReplicationQueues

Parameters:

queueId - a String that identifies the queue.

filename - name of the WAL

Throws:

ReplicationException

removeLog

public void removeLog(String queueId,
                      String filename)

Description copied from interface: ReplicationQueues

Remove an WAL file from the given queue.

Specified by:

removeLog in interface ReplicationQueues

Parameters:

queueId - a String that identifies the queue.

filename - name of the WAL

setLogPosition

public void setLogPosition(String queueId,
                           String filename,
                           long position)

Description copied from interface: ReplicationQueues

Set the current position for a specific WAL in a given queue.

Specified by:

setLogPosition in interface ReplicationQueues

Parameters:

queueId - a String that identifies the queue

filename - name of the WAL

position - the current position in the file

getLogPosition

public long getLogPosition(String queueId,
                           String filename)
                    throws ReplicationException

Description copied from interface: ReplicationQueues

Get the current position for a specific WAL in a given queue.

Specified by:

getLogPosition in interface ReplicationQueues

Parameters:

queueId - a String that identifies the queue

filename - name of the WAL

Returns:

the current position in the file

Throws:

ReplicationException

isThisOurRegionServer

public boolean isThisOurRegionServer(String regionserver)

Description copied from interface: ReplicationQueues

Checks if the provided znode is the same as this region server's

Specified by:

isThisOurRegionServer in interface ReplicationQueues

Parameters:

regionserver - the id of the region server

Returns:

if this is this rs's znode

getUnClaimedQueueIds

public List<String> getUnClaimedQueueIds(String regionserver)

Description copied from interface: ReplicationQueues

Get queueIds from a dead region server, whose queues has not been claimed by other region servers.

Specified by:

getUnClaimedQueueIds in interface ReplicationQueues

Returns:

empty if the queue exists but no children, null if the queue does not exist.

claimQueue

public Pair<String,SortedSet<String>> claimQueue(String regionserver,
                                                 String queueId)

Description copied from interface: ReplicationQueues

Take ownership for the queue identified by queueId and belongs to a dead region server.

Specified by:

claimQueue in interface ReplicationQueues

Parameters:

regionserver - the id of the dead region server

queueId - the id of the queue

Returns:

the new PeerId and A SortedSet of WALs in its queue, and null if no unclaimed queue.

removeReplicatorIfQueueIsEmpty

public void removeReplicatorIfQueueIsEmpty(String regionserver)

Description copied from interface: ReplicationQueues

Remove the znode of region server if the queue is empty.

Specified by:

removeReplicatorIfQueueIsEmpty in interface ReplicationQueues

removeAllQueues

public void removeAllQueues()

Description copied from interface: ReplicationQueues

Remove all replication queues for this region server.

Specified by:

removeAllQueues in interface ReplicationQueues

getLogsInQueue

public List<String> getLogsInQueue(String queueId)

Description copied from interface: ReplicationQueues

Get a list of all WALs in the given queue.

Specified by:

getLogsInQueue in interface ReplicationQueues

Parameters:

queueId - a String that identifies the queue

Returns:

a list of WALs, null if no such queue exists for this server

getAllQueues

public List<String> getAllQueues()

Description copied from interface: ReplicationQueues

Get a list of all queues for this region server.

Specified by:

getAllQueues in interface ReplicationQueues

Returns:

a list of queueIds, an empty list if this region server is dead and has no outstanding queues

moveQueueUsingMulti

private Pair<String,SortedSet<String>> moveQueueUsingMulti(String znode,
                                                           String peerId)

It "atomically" copies one peer's wals queue from another dead region server and returns them all sorted. The new peer id is equal to the old peer id appended with the dead server's znode.

Parameters:

znode - pertaining to the region server to copy the queues from

addHFileRefs

public void addHFileRefs(String peerId,
                         List<Pair<org.apache.hadoop.fs.Path,org.apache.hadoop.fs.Path>> pairs)
                  throws ReplicationException

Description copied from interface: ReplicationQueues

Add new hfile references to the queue.

Specified by:

addHFileRefs in interface ReplicationQueues

Parameters:

peerId - peer cluster id to which the hfiles need to be replicated

pairs - list of pairs of { HFile location in staging dir, HFile path in region dir which will be added in the queue }

Throws:

ReplicationException - if fails to add a hfile reference

removeHFileRefs

public void removeHFileRefs(String peerId,
                            List<String> files)

Description copied from interface: ReplicationQueues

Remove hfile references from the queue.

Specified by:

removeHFileRefs in interface ReplicationQueues

Parameters:

peerId - peer cluster id from which this hfile references needs to be removed

files - list of hfile references to be removed

addPeerToHFileRefs

public void addPeerToHFileRefs(String peerId)
                        throws ReplicationException

Description copied from interface: ReplicationQueues

Add a peer to hfile reference queue if peer does not exist.

Specified by:

addPeerToHFileRefs in interface ReplicationQueues

Parameters:

peerId - peer cluster id to be added

Throws:

ReplicationException - if fails to add a peer id to hfile reference queue

removePeerFromHFileRefs

public void removePeerFromHFileRefs(String peerId)

Description copied from interface: ReplicationQueues

Remove a peer from hfile reference queue.

Specified by:

removePeerFromHFileRefs in interface ReplicationQueues

Parameters:

peerId - peer cluster id to be removed