Package org.apache.hadoop.hbase.backup.replication

Class ContinuousBackupReplicationEndpoint

java.lang.Object

org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService

org.apache.hadoop.hbase.replication.BaseReplicationEndpoint

org.apache.hadoop.hbase.backup.replication.ContinuousBackupReplicationEndpoint

All Implemented Interfaces:

ReplicationEndpoint, ReplicationPeerConfigListener, org.apache.hbase.thirdparty.com.google.common.util.concurrent.Service

@Private
public class ContinuousBackupReplicationEndpoint
extends BaseReplicationEndpoint

ContinuousBackupReplicationEndpoint is responsible for replicating WAL entries to a backup storage. It organizes WAL entries by day and periodically flushes the data, ensuring that WAL files do not exceed the configured size. The class includes mechanisms for handling the WAL files, performing bulk load backups, and ensuring that the replication process is safe.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hadoop.hbase.replication.ReplicationEndpoint

ReplicationEndpoint.Context, ReplicationEndpoint.ReplicateContext

Nested classes/interfaces inherited from interface org.apache.hbase.thirdparty.com.google.common.util.concurrent.Service

org.apache.hbase.thirdparty.com.google.common.util.concurrent.Service.Listener, org.apache.hbase.thirdparty.com.google.common.util.concurrent.Service.State

Field Summary

Fields

Modifier and Type	Field	Description
private BackupFileSystemManager	backupFileSystemManager
private org.apache.hadoop.conf.Configuration	conf
static final String	CONF_BACKUP_ROOT_DIR
static final String	CONF_PEER_UUID
private long	latestWALEntryTimestamp
private final ReentrantLock	lock
private static final org.slf4j.Logger	LOG
static final long	ONE_DAY_IN_MILLISECONDS
private String	peerId
private UUID	peerUUID
private ReplicationSourceInterface	replicationSource
static final String	WAL_FILE_PREFIX
private final Map<Long,FSHLogProvider.Writer>	walWriters

Fields inherited from class org.apache.hadoop.hbase.replication.BaseReplicationEndpoint

ctx, REPLICATION_WALENTRYFILTER_CONFIG_KEY

Constructor Summary

Constructors

Constructor	Description
ContinuousBackupReplicationEndpoint()

Method Summary

Modifier and Type	Method	Description
private void	backupWalEntries(long day, List<WAL.Entry> walEntries)
void	beforePersistingReplicationOffset()	Hook invoked before persisting replication offsets.
private void	close()
(package private) static void	copyWithCleanup(org.apache.hadoop.fs.FileSystem srcFS, org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.FileSystem dstFS, org.apache.hadoop.fs.Path dst, org.apache.hadoop.conf.Configuration conf)	Copy a file with cleanup logic in case of failure.
private FSHLogProvider.Writer	createWalWriter(long dayInMillis)
protected void	doStart()
protected void	doStop()
private static org.apache.hadoop.fs.Path	findExistingPath(org.apache.hadoop.fs.FileSystem rootFs, org.apache.hadoop.fs.Path baseNamespaceDir, org.apache.hadoop.fs.Path hFileArchiveDir, org.apache.hadoop.fs.Path filePath)
private void	flushWriters()
private org.apache.hadoop.fs.Path	getBulkLoadFileStagingPath(org.apache.hadoop.fs.Path relativePathFromNamespace)
UUID	getPeerUUID()	Returns a UUID of the provided peer id.
private Map<Long,List<WAL.Entry>>	groupEntriesByDay(List<WAL.Entry> entries)
void	init(ReplicationEndpoint.Context context)	Initialize the replication endpoint with the given context.
private void	initializeBackupFileSystemManager()
private void	initializePeerUUID()
boolean	replicate(ReplicationEndpoint.ReplicateContext replicateContext)	Replicate the given set of entries (in the context) to the other cluster.
void	start()	Initiates service startup and returns immediately.
void	stop()	If the service is starting or running, this initiates service shutdown and returns immediately.
private void	updateLastReplicatedTimestampForContinuousBackup()	Persists the latest replicated WAL entry timestamp in the backup system table.
(package private) void	uploadBulkLoadFiles(long dayInMillis, List<org.apache.hadoop.fs.Path> bulkLoadFiles)

Methods inherited from class org.apache.hadoop.hbase.replication.BaseReplicationEndpoint

canReplicateToSameCluster, getNamespaceTableCfWALEntryFilter, getScopeWALEntryFilter, getWALEntryfilter, isStarting, peerConfigUpdated

Methods inherited from class org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService

addListener, awaitRunning, awaitRunning, awaitRunning, awaitTerminated, awaitTerminated, awaitTerminated, doCancelStart, failureCause, isRunning, notifyFailed, notifyStarted, notifyStopped, startAsync, state, stopAsync, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.hadoop.hbase.replication.ReplicationEndpoint

awaitRunning, awaitRunning, awaitTerminated, awaitTerminated, failureCause, isRunning

Field Details

LOG

private static final org.slf4j.Logger LOG

CONF_PEER_UUID

public static final String CONF_PEER_UUID

See Also:

• Constant Field Values

CONF_BACKUP_ROOT_DIR

public static final String CONF_BACKUP_ROOT_DIR

See Also:

• Constant Field Values

walWriters

private final Map<Long,FSHLogProvider.Writer> walWriters

lock

private final ReentrantLock lock

replicationSource

private ReplicationSourceInterface replicationSource

conf

private org.apache.hadoop.conf.Configuration conf

backupFileSystemManager

private BackupFileSystemManager backupFileSystemManager

peerUUID

private UUID peerUUID

peerId

private String peerId

latestWALEntryTimestamp

private long latestWALEntryTimestamp

ONE_DAY_IN_MILLISECONDS

public static final long ONE_DAY_IN_MILLISECONDS

WAL_FILE_PREFIX

public static final String WAL_FILE_PREFIX

See Also:

• Constant Field Values

Constructor Details

ContinuousBackupReplicationEndpoint

public ContinuousBackupReplicationEndpoint()

Method Details

init

public void init(ReplicationEndpoint.Context context)
          throws IOException

Description copied from interface: ReplicationEndpoint

Initialize the replication endpoint with the given context.

Specified by:

init in interface ReplicationEndpoint

Overrides:

init in class BaseReplicationEndpoint

Parameters:

context - replication context

Throws:

IOException - error occur when initialize the endpoint.

initializePeerUUID

private void initializePeerUUID()
                         throws IOException

Throws:

IOException

initializeBackupFileSystemManager

private void initializeBackupFileSystemManager()
                                        throws IOException

Throws:

IOException

flushWriters

private void flushWriters()
                   throws IOException

Throws:

IOException

getPeerUUID

public UUID getPeerUUID()

Description copied from interface: ReplicationEndpoint

Returns a UUID of the provided peer id. Every HBase cluster instance has a persisted associated UUID. If the replication is not performed to an actual HBase cluster (but some other system), the UUID returned has to uniquely identify the connected target system.

Returns:

a UUID or null if the peer cluster does not exist or is not connected.

start

public void start()

Description copied from interface: ReplicationEndpoint

Initiates service startup and returns immediately. A stopped service may not be restarted. Equivalent of startAsync call in Guava Service.

doStart

protected void doStart()

Specified by:

doStart in class org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService

replicate

public boolean replicate(ReplicationEndpoint.ReplicateContext replicateContext)

Description copied from interface: ReplicationEndpoint

Replicate the given set of entries (in the context) to the other cluster. Can block until all the given entries are replicated. Upon this method is returned, all entries that were passed in the context are assumed to be persisted in the target cluster.

Parameters:

replicateContext - a context where WAL entries and other parameters can be obtained.

updateLastReplicatedTimestampForContinuousBackup

private void updateLastReplicatedTimestampForContinuousBackup()
                                                       throws IOException

Persists the latest replicated WAL entry timestamp in the backup system table. This checkpoint is critical for Continuous Backup and Point-in-Time Restore (PITR) to ensure restore operations only go up to a known safe point. The value is stored per region server using its ServerName as the key.

Throws:

IOException - if the checkpoint update fails

groupEntriesByDay

private Map<Long,List<WAL.Entry>> groupEntriesByDay(List<WAL.Entry> entries)

backupWalEntries

private void backupWalEntries(long day,
 List<WAL.Entry> walEntries)
                       throws IOException

Throws:

IOException

createWalWriter

private FSHLogProvider.Writer createWalWriter(long dayInMillis)

stop

public void stop()

Description copied from interface: ReplicationEndpoint

If the service is starting or running, this initiates service shutdown and returns immediately. If the service has already been stopped, this method returns immediately without taking action. Equivalent of stopAsync call in Guava Service.

doStop

protected void doStop()

Specified by:

doStop in class org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService

close

private void close()

uploadBulkLoadFiles

void uploadBulkLoadFiles(long dayInMillis,
 List<org.apache.hadoop.fs.Path> bulkLoadFiles)
                  throws BulkLoadUploadException

Throws:

BulkLoadUploadException

copyWithCleanup

static void copyWithCleanup(org.apache.hadoop.fs.FileSystem srcFS,
 org.apache.hadoop.fs.Path src,
 org.apache.hadoop.fs.FileSystem dstFS,
 org.apache.hadoop.fs.Path dst,
 org.apache.hadoop.conf.Configuration conf)
                     throws IOException

Copy a file with cleanup logic in case of failure. Always overwrite destination to avoid leaving corrupt partial files.

Throws:

IOException

getBulkLoadFileStagingPath

private org.apache.hadoop.fs.Path getBulkLoadFileStagingPath(org.apache.hadoop.fs.Path relativePathFromNamespace)
                                                      throws IOException

Throws:

IOException

findExistingPath

private static org.apache.hadoop.fs.Path findExistingPath(org.apache.hadoop.fs.FileSystem rootFs,
 org.apache.hadoop.fs.Path baseNamespaceDir,
 org.apache.hadoop.fs.Path hFileArchiveDir,
 org.apache.hadoop.fs.Path filePath)
                                                   throws IOException

Throws:

IOException

beforePersistingReplicationOffset

public void beforePersistingReplicationOffset()
                                       throws IOException

Description copied from interface: ReplicationEndpoint

Hook invoked before persisting replication offsets. Eg: Buffered endpoints can flush/close WALs here.

Throws:

IOException