Interface ReplicationQueueStorage
- All Known Implementing Classes:
OfflineTableReplicationQueueStorage
,TableReplicationQueueStorage
Perform read/write to the replication queue storage.
-
Method Summary
Modifier and TypeMethodDescriptionvoid
addHFileRefs
(String peerId, List<Pair<org.apache.hadoop.fs.Path, org.apache.hadoop.fs.Path>> pairs) Add new hfile references to the queue.void
batchUpdateHFileRefs
(String peerId, List<String> hfileRefs) Add the given hfile refs to the given peer.void
batchUpdateLastSequenceIds
(List<ZKReplicationQueueStorageForMigration.ZkLastPushedSeqId> lastPushedSeqIds) Update last pushed sequence id for the given regions and peers.void
batchUpdateQueues
(ServerName serverName, List<ReplicationQueueData> datas) Update the replication queue datas for a given region server.claimQueue
(ReplicationQueueId queueId, ServerName targetServerName) Change ownership for the queue identified by queueId and belongs to a dead region server.Load all hfile references in all replication queues.Get list of all peers from hfile reference queue.long
getLastSequenceId
(String encodedRegionName, String peerId) Read the max sequence id of the specific region for a given peer.getOffsets
(ReplicationQueueId queueId) Get the current offset of all the WAL groups for a queuegetReplicableHFiles
(String peerId) Get a list of all hfile references in the given peer.boolean
hasData()
Whether the replication queue table exists.listAllQueueIds
(String peerId) Get a list of all queues for the specific peer.listAllQueueIds
(String peerId, ServerName serverName) Get a list of all queues for the specific region server and the specific peerlistAllQueueIds
(ServerName serverName) Get a list of all queues for the specific region server.Get a list of all queues and the offsets.Get a list of all region servers that have outstanding replication queues.void
removeAllQueues
(String peerId) Remove all the replication queues for the given peer.void
removeHFileRefs
(String peerId, List<String> files) Remove hfile references from the queue.void
removeLastSequenceIds
(String peerId) Remove all the max sequence id record for the given peer.void
removeLastSequenceIds
(String peerId, List<String> encodedRegionNames) Remove the max sequence id record for the given peer and regions.void
Remove all the last sequence ids and hfile references data which are written before the given timestamp.void
removePeerFromHFileRefs
(String peerId) Remove a peer from hfile reference queue.void
removeQueue
(ReplicationQueueId queueId) Remove a replication queuevoid
setLastSequenceIds
(String peerId, Map<String, Long> lastSeqIds) Set the max sequence id of a bunch of regions for a given peer.void
setOffset
(ReplicationQueueId queueId, String walGroup, ReplicationGroupOffset offset, Map<String, Long> lastSeqIds) Set the current offset for a specific WAL group in a given queue.
-
Method Details
-
setOffset
void setOffset(ReplicationQueueId queueId, String walGroup, ReplicationGroupOffset offset, Map<String, Long> lastSeqIds) throws ReplicationExceptionSet the current offset for a specific WAL group in a given queue.- Parameters:
queueId
- the id of the queuewalGroup
- the group of the WAL, can be empty if multi wal is not enabledoffset
- the current offset of replication progresslastSeqIds
- map with {encodedRegionName, sequenceId} pairs for serial replication.- Throws:
ReplicationException
-
getOffsets
Map<String,ReplicationGroupOffset> getOffsets(ReplicationQueueId queueId) throws ReplicationException Get the current offset of all the WAL groups for a queue- Parameters:
queueId
- the id of the queue- Returns:
- a map of all offsets of the WAL groups. The key the is WAL group and the value is the position.
- Throws:
ReplicationException
-
listAllQueueIds
Get a list of all queues for the specific peer.- Parameters:
peerId
- the id of the peer- Returns:
- a list of queueIds
- Throws:
ReplicationException
-
listAllQueueIds
Get a list of all queues for the specific region server.- Parameters:
serverName
- the server name of the region server that owns the set of queues- Returns:
- a list of queueIds
- Throws:
ReplicationException
-
listAllQueueIds
List<ReplicationQueueId> listAllQueueIds(String peerId, ServerName serverName) throws ReplicationException Get a list of all queues for the specific region server and the specific peer- Parameters:
peerId
- the id of the peerserverName
- the server name of the region server that owns the set of queues- Returns:
- a list of queueIds
- Throws:
ReplicationException
-
listAllQueues
Get a list of all queues and the offsets.- Throws:
ReplicationException
-
listAllReplicators
Get a list of all region servers that have outstanding replication queues. These servers could be alive, dead or from a previous run of the cluster.- Returns:
- a list of server names
- Throws:
ReplicationException
-
claimQueue
Map<String,ReplicationGroupOffset> claimQueue(ReplicationQueueId queueId, ServerName targetServerName) throws ReplicationException Change ownership for the queue identified by queueId and belongs to a dead region server.- Parameters:
queueId
- the id of the queuetargetServerName
- the name of the target region server- Returns:
- the new PeerId and A SortedSet of WALs in its queue
- Throws:
ReplicationException
-
removeQueue
Remove a replication queue- Parameters:
queueId
- the id of the queue to remove- Throws:
ReplicationException
-
removeAllQueues
Remove all the replication queues for the given peer. Usually used when removing a peer.- Parameters:
peerId
- the id of the peer- Throws:
ReplicationException
-
getLastSequenceId
Read the max sequence id of the specific region for a given peer. For serial replication, we need the max sequenced id to decide whether we can push the next entries.- Parameters:
encodedRegionName
- the encoded region namepeerId
- peer id- Returns:
- the max sequence id of the specific region for a given peer.
- Throws:
ReplicationException
-
setLastSequenceIds
Set the max sequence id of a bunch of regions for a given peer. Will be called when setting up a serial replication peer.- Parameters:
peerId
- peer idlastSeqIds
- map with {encodedRegionName, sequenceId} pairs for serial replication.- Throws:
ReplicationException
-
removeLastSequenceIds
Remove all the max sequence id record for the given peer.- Parameters:
peerId
- peer id- Throws:
ReplicationException
-
removeLastSequenceIds
void removeLastSequenceIds(String peerId, List<String> encodedRegionNames) throws ReplicationException Remove the max sequence id record for the given peer and regions.- Parameters:
peerId
- peer idencodedRegionNames
- the encoded region names- Throws:
ReplicationException
-
removePeerFromHFileRefs
Remove a peer from hfile reference queue.- Parameters:
peerId
- peer cluster id to be removed- Throws:
ReplicationException
-
addHFileRefs
void addHFileRefs(String peerId, List<Pair<org.apache.hadoop.fs.Path, org.apache.hadoop.fs.Path>> pairs) throws ReplicationExceptionAdd new hfile references to the queue.- Parameters:
peerId
- peer cluster id to which the hfiles need to be replicatedpairs
- list of pairs of { HFile location in staging dir, HFile path in region dir which will be added in the queue }- Throws:
ReplicationException
- if fails to add a hfile reference
-
removeHFileRefs
Remove hfile references from the queue.- Parameters:
peerId
- peer cluster id from which this hfile references needs to be removedfiles
- list of hfile references to be removed- Throws:
ReplicationException
-
getAllPeersFromHFileRefsQueue
Get list of all peers from hfile reference queue.- Returns:
- a list of peer ids
- Throws:
ReplicationException
-
getReplicableHFiles
Get a list of all hfile references in the given peer.- Parameters:
peerId
- a String that identifies the peer- Returns:
- a list of hfile references
- Throws:
ReplicationException
-
getAllHFileRefs
Load all hfile references in all replication queues. This method guarantees to return a snapshot which contains all hfile references at the start of this call. However, some newly created hfile references during the call may not be included.- Throws:
ReplicationException
-
hasData
Whether the replication queue table exists.- Returns:
- Whether the replication queue table exists
- Throws:
ReplicationException
-
batchUpdateQueues
void batchUpdateQueues(ServerName serverName, List<ReplicationQueueData> datas) throws ReplicationException Update the replication queue datas for a given region server.- Throws:
ReplicationException
-
batchUpdateLastSequenceIds
void batchUpdateLastSequenceIds(List<ZKReplicationQueueStorageForMigration.ZkLastPushedSeqId> lastPushedSeqIds) throws ReplicationException Update last pushed sequence id for the given regions and peers.- Throws:
ReplicationException
-
batchUpdateHFileRefs
Add the given hfile refs to the given peer.- Throws:
ReplicationException
-
removeLastSequenceIdsAndHFileRefsBefore
Remove all the last sequence ids and hfile references data which are written before the given timestamp. The data of these two types are not used by replication directly. For last sequence ids, we will check it in serial replication, to make sure that we will replicate all edits in order, so if there are stale data, the worst case is that we will stop replicating as we think we still need to finish previous ranges first, although actually we have already replicated them out. For hfile references, it is just used by hfile cleaner to not remove these hfiles before we replicate them out, so if there are stale data, the worst case is that we can not remove these hfiles, although actually they have already been replicated out. So it is OK for us to just bring up the cluster first, and then use this method to delete the stale data, i.e, the data which are written before a specific timestamp.- Throws:
ReplicationException
-