Package org.apache.hadoop.hbase.mapreduce.replication

Class VerifyReplication

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, org.apache.hadoop.util.Tool

@Private
public class VerifyReplication
extends org.apache.hadoop.conf.Configured
implements org.apache.hadoop.util.Tool

This map-only job compares the data from a local table with a remote one. Every cell is compared and must have exactly the same keys (even timestamp) as well as same value. It is possible to restrict the job by time range and families. The peer id that's provided must match the one given when the replication stream was setup.

Two counters are provided, Verifier.Counters.GOODROWS and BADROWS. The reason for a why a row is different is shown in the map's log.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	VerifyReplication.Verifier	Map-only comparator for 2 tables

Field Summary

Fields

Modifier and Type	Field	Description
(package private) int	batch
(package private) String	delimiter
(package private) long	endTime
(package private) String	families
(package private) boolean	includeDeletedCells
private static final String	JOB_NAME_CONF_KEY
private static final org.slf4j.Logger	LOG
static final String	NAME
private static final String	PEER_CONFIG_PREFIX
(package private) String	peerFSAddress
(package private) String	peerHBaseRootAddress
(package private) String	peerId
(package private) String	peerQuorumAddress
(package private) String	peerSnapshotName
(package private) String	peerSnapshotTmpDir
(package private) String	peerTableName
(package private) int	reCompareBackoffExponent
private static ThreadPoolExecutor	reCompareExecutor
(package private) int	reCompareThreads
(package private) int	reCompareTries
(package private) String	rowPrefixes
(package private) int	sleepMsBeforeReCompare
(package private) String	sourceSnapshotName
(package private) String	sourceSnapshotTmpDir
(package private) long	startTime
(package private) String	tableName
(package private) boolean	verbose
(package private) int	versions

Constructor Summary

Constructors

Constructor	Description
VerifyReplication()

Method Summary

Modifier and Type	Method	Description
private org.apache.hadoop.conf.Configuration	applyURIConf(org.apache.hadoop.conf.Configuration conf, URI uri)
private static ThreadPoolExecutor	buildReCompareExecutor(int maxThreads, org.apache.hadoop.mapreduce.Mapper.Context context)
private static ThreadPoolExecutor.CallerRunsPolicy	buildRejectedReComparePolicy(org.apache.hadoop.mapreduce.Mapper.Context context)
org.apache.hadoop.mapreduce.Job	createSubmittableJob(org.apache.hadoop.conf.Configuration conf, String[] args)	Sets up the actual job.
boolean	doCommandLine(String[] args)
private static Pair<ReplicationPeerConfig,org.apache.hadoop.conf.Configuration>	getPeerQuorumConfig(org.apache.hadoop.conf.Configuration conf, String peerId)
protected static byte[]	getRow(Result sourceResult, Result replicatedResult)
private boolean	isPeerQuorumAddress(String cmd)
static void	main(String[] args)	Main entry point.
private static void	printUsage(String errorMsg)
private void	restoreSnapshotForPeerCluster(org.apache.hadoop.conf.Configuration conf, String peerQuorumAddress)
int	run(String[] args)
private static void	setRowPrefixFilter(Scan scan, String rowPrefixes)
private static void	setStartAndStopRows(Scan scan, byte[] startPrefixRow, byte[] lastPrefixRow)

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.conf.Configurable

getConf, setConf

Field Details

LOG

private static final org.slf4j.Logger LOG

NAME

public static final String NAME

See Also:

• Constant Field Values

PEER_CONFIG_PREFIX

private static final String PEER_CONFIG_PREFIX

See Also:

• Constant Field Values

reCompareExecutor

private static ThreadPoolExecutor reCompareExecutor

reCompareTries

int reCompareTries

reCompareBackoffExponent

int reCompareBackoffExponent

reCompareThreads

int reCompareThreads

sleepMsBeforeReCompare

int sleepMsBeforeReCompare

startTime

long startTime

endTime

long endTime

batch

int batch

versions

int versions

tableName

String tableName

families

String families

delimiter

String delimiter

peerId

String peerId

peerQuorumAddress

String peerQuorumAddress

rowPrefixes

String rowPrefixes

verbose

boolean verbose

includeDeletedCells

boolean includeDeletedCells

sourceSnapshotName

String sourceSnapshotName

sourceSnapshotTmpDir

String sourceSnapshotTmpDir

peerSnapshotName

String peerSnapshotName

peerSnapshotTmpDir

String peerSnapshotTmpDir

peerFSAddress

String peerFSAddress

peerHBaseRootAddress

String peerHBaseRootAddress

peerTableName

String peerTableName

JOB_NAME_CONF_KEY

private static final String JOB_NAME_CONF_KEY

See Also:

• Constant Field Values

Constructor Details

VerifyReplication

public VerifyReplication()

Method Details

getPeerQuorumConfig

private static Pair<ReplicationPeerConfig,org.apache.hadoop.conf.Configuration> getPeerQuorumConfig(org.apache.hadoop.conf.Configuration conf,
 String peerId)
                                                                                             throws IOException

Throws:

IOException

applyURIConf

private org.apache.hadoop.conf.Configuration applyURIConf(org.apache.hadoop.conf.Configuration conf,
 URI uri)

restoreSnapshotForPeerCluster

private void restoreSnapshotForPeerCluster(org.apache.hadoop.conf.Configuration conf,
 String peerQuorumAddress)
                                    throws IOException

Throws:

IOException

createSubmittableJob

public org.apache.hadoop.mapreduce.Job createSubmittableJob(org.apache.hadoop.conf.Configuration conf,
 String[] args)
                                                     throws IOException

Sets up the actual job.

Parameters:

conf - The current configuration.

args - The command line parameters.

Returns:

The newly created job.

Throws:

IOException - When setting up the job fails.

getRow

protected static byte[] getRow(Result sourceResult,
 Result replicatedResult)

setRowPrefixFilter

private static void setRowPrefixFilter(Scan scan,
 String rowPrefixes)

setStartAndStopRows

private static void setStartAndStopRows(Scan scan,
 byte[] startPrefixRow,
 byte[] lastPrefixRow)

doCommandLine

public boolean doCommandLine(String[] args)

isPeerQuorumAddress

private boolean isPeerQuorumAddress(String cmd)

printUsage

private static void printUsage(String errorMsg)

buildReCompareExecutor

private static ThreadPoolExecutor buildReCompareExecutor(int maxThreads,
 org.apache.hadoop.mapreduce.Mapper.Context context)

buildRejectedReComparePolicy

private static ThreadPoolExecutor.CallerRunsPolicy buildRejectedReComparePolicy(org.apache.hadoop.mapreduce.Mapper.Context context)

run

public int run(String[] args)
        throws Exception

Specified by:

run in interface org.apache.hadoop.util.Tool

Throws:

Exception

main

public static void main(String[] args)
                 throws Exception

Main entry point.

Parameters:

args - The command line parameters.

Throws:

Exception - When running the job fails.