Read Replica Cluster

Background

A Read Replica Cluster is an entire HBase cluster running in global read-only mode against the same shared storage (hbase.rootdir) as an active read-write cluster. Both clusters list the same HFiles in the same HDFS / cloud-object-store location; no data is copied. Reads can be served from either cluster, letting the read workload be fanned out across multiple clusters without doubling storage cost.

Typical use cases:

• Fan out heavy scan / analytical workloads off the primary cluster.
• Add cross-availability-zone read capacity backed by a single shared bucket.
• Stand up an isolated cluster for read-mostly experiments without copying data.

The parent design lives on HBASE-29081.

Design

The feature has three parts.

Custom hbase:meta per cluster

Every cluster sharing a hbase.rootdir needs its own hbase:meta and its own master local region directory, because region assignments and master-local state are node-scoped and cannot be shared. Other system tables (hbase:acl, hbase:replication) are safe to share because their contents are storage-wide and the replica never writes to them.

The configuration key hbase.meta.table.suffix selects a per-cluster suffix; the meta table becomes hbase:meta_<suffix> and the master's local store directory becomes MasterData_<suffix>. Each cluster sharing the same hbase.rootdir must be configured with a distinct suffix so its hbase:meta and MasterData directory do not collide with any other cluster's. The suffix must match [a-zA-Z0-9]+.

Global read-only mode

hbase.global.readonly.enabled=true puts a cluster into read-only mode. Five coprocessor controllers under org.apache.hadoop.hbase.security.access intercept every user-table mutation path and throw WriteAttemptedOnReadOnlyClusterException (a DoNotRetryIOException) with the message Operation not allowed in Read-Only Mode:

Operators do not load these classes manually. CoprocessorConfigurationUtil.syncReadOnlyConfigurations adds them to hbase.coprocessor.master.classes, hbase.coprocessor.regionserver.classes, and hbase.coprocessor.region.classes at startup and on every dynamic ConfigurationManager.notifyAllObservers event — so the flag can be flipped at runtime with update_all_config (see Case 3).

Preventing Multiple Active Clusters (active.cluster.suffix.id)

Two clusters writing to the same hbase.rootdir would corrupt shared storage. To enforce a single writer, an active master creates a protobuf-serialized sentinel at <hbase.rootdir>/active.cluster.suffix.id recording its cluster ID and meta suffix. MasterFileSystem.negotiateActiveClusterSuffixFile runs at master startup:

• An active cluster (hbase.global.readonly.enabled=false) creates the file if absent, or verifies its contents match its own identity. If the file belongs to another cluster, startup aborts with an IOException.
• A replica cluster (hbase.global.readonly.enabled=true) does not read or write the file; it logs [Read-replica feature] Replica cluster is being started in Read Only Mode and continues.

AbstractReadOnlyController.manageActiveClusterIdFile handles the dynamic toggle: switching to read-only deletes the file if this cluster owns it, and switching back to read-write creates the file if absent.

Configuration

On every node of the read replica cluster, add the following to hbase-site.xml:

<property>
  <name>hbase.global.readonly.enabled</name>
  <value>true</value>
  <description>
    Put this cluster into global read-only mode. All user-table writes, flushes,
    compactions, splits, and merges are blocked. The five ReadOnly coprocessor
    controllers are loaded automatically.
  </description>
</property>
<property>
  <name>hbase.meta.table.suffix</name>
  <value>replica1</value>
  <description>
    Optional. If set, the meta table is named hbase:meta_<suffix> and the
    master's local store directory is MasterData_<suffix>. Value must match
    [a-zA-Z0-9]+. Each cluster sharing the same hbase.rootdir MUST be
    configured with a distinct suffix so its hbase:meta and MasterData
    directory do not collide with any other cluster's.
  </description>
</property>

The active cluster uses the same hbase.rootdir but its own hbase.meta.table.suffix (distinct from every replica's suffix), and leaves hbase.global.readonly.enabled unset or false.

hbase.global.readonly.enabled is a dynamic configuration — a config-change event reloads the read-only coprocessors without restarting the process. All nodes must agree on the value; operators are responsible for keeping every hbase-site.xml in sync before issuing update_all_config.

Operation and maintenance

Case 1. Bring up a new read replica cluster

1. Provision the replica cluster on hardware that can reach the active cluster's hbase.rootdir (typically the same HDFS or object store).
2. Set hbase.global.readonly.enabled=true in the replica's hbase-site.xml. And set up hbase.meta.table.suffix, to distinguish the replica cluster's meta table on the shared storage.
3. Start the cluster and verify if the Master log shows [Read-replica feature] Replica cluster is being started in Read Only Mode.
4. From the replica shell, run refresh_meta and then refresh_hfiles to materialize the active cluster's current state on the replica.

Case 2. Routine sync after writes on the active cluster

# On the active cluster
hbase> flush 'my_namespace:my_table'

# On the read replica cluster
hbase> refresh_meta
hbase> refresh_hfiles 'TABLE_NAME' => 'my_namespace:my_table'

Always run refresh_meta first, then refresh_hfiles. refresh_hfiles only refreshes regions that are open on the replica, so newly discovered regions must be in meta (and assigned) before their HFiles can be picked up. refresh_hfiles supports three scopes:

hbase> refresh_hfiles                              # all user tables
hbase> refresh_hfiles 'TABLE_NAME' => 'ns:table'   # one table
hbase> refresh_hfiles 'NAMESPACE'  => 'ns'         # one namespace

Passing both TABLE_NAME and NAMESPACE to refresh_hfiles is rejected. Both commands return a procedure ID that can be tracked through the master UI or Admin.getProcedures().

If the replica's block cache holds stale entries for a table that has just been refreshed, evict them with the pre-existing clear_block_cache 'my_namespace:my_table' shell command.

Admin and AsyncAdmin expose the same operations programmatically:

long pid;
pid = admin.refreshMeta();
pid = admin.refreshHFiles();                              // all user tables
pid = admin.refreshHFiles(TableName.valueOf("ns:table")); // one table
pid = admin.refreshHFiles("ns");                          // one namespace

Case 3. Dynamically toggle read-only mode

hbase.global.readonly.enabled can be changed without a restart. Edit hbase-site.xml then trigger a configuration refresh (for example, update_all_config from the shell). ConfigurationManager notifies its observers, which load or unload the read-only coprocessors and call AbstractReadOnlyController.manageActiveClusterIdFile:

• false → true (becoming a replica): the active.cluster.suffix.id file is deleted only if its contents match this cluster; if another cluster owns the file, it is left in place.
• true → false (becoming active): the file is recreated with this cluster's identity, unless it already exists.

In-flight batch operations are not interrupted; write operations submitted after the toggle throw WriteAttemptedOnReadOnlyClusterException. The caller is responsible for handling and (if desired) resubmitting the failed mutations.

Case 4. Promote a replica when the active cluster is lost

1. Confirm the original active cluster is fully down.
2. If a stale active.cluster.suffix.id from the previous active is still present, remove it manually (e.g. hdfs dfs -rm <hbase.rootdir>/active.cluster.suffix.id, or the equivalent CLI for your object store). The new active master will refuse to start while a foreign sentinel file is in place.
3. Set hbase.global.readonly.enabled=false on the replica and apply the change (dynamic update or restart). The master writes a fresh sentinel file with this cluster's identity.

Case 5. Recover from a blocked read-only transition

If you attempt to promote a replica to active (hbase.global.readonly.enabled=false + update_all_config) while a different cluster's active.cluster.suffix.id file is still present, the transition is blocked. The shell returns a ReadOnlyTransitionException and the cluster remains in read-only mode — writes continue to be rejected with WriteAttemptedOnReadOnlyClusterException.

The server ERROR log includes the foreign cluster's ID:

Cannot disable read-only mode. The active.cluster.suffix.id file contains a different
cluster ID (<foreign-id>), which means that cluster is already the active cluster.
Reverting hbase.global.readonly.enabled to true.

To recover, choose one of the following paths depending on whether the existing active cluster is still running.

Path A — the active cluster is still running:

1. On the currently active cluster, set hbase.global.readonly.enabled=true in hbase-site.xml and run update_all_config. This converts it to a replica and deletes its active.cluster.suffix.id file.
2. On the cluster you want to promote, re-run update_all_config. The transition will now succeed — the cluster writes a fresh sentinel file with its own identity and unloads the read-only coprocessors.

Path B — the active cluster is down or has already been converted to a replica:

1. Confirm the cluster that owns the sentinel file is fully stopped or has already been converted to a replica.
2. Remove the foreign sentinel file:

   hdfs dfs -rm <hbase.rootdir>/active.cluster.suffix.id

3. On the cluster you want to promote, re-run update_all_config. The transition will now succeed — the cluster writes a fresh sentinel file with its own identity and unloads the read-only coprocessors.

Configurations and Commands

New configs

New commands