org.apache.hadoop.hbase.util

Class MultiThreadedWriterBase

java.lang.Object

org.apache.hadoop.hbase.util.MultiThreadedAction

org.apache.hadoop.hbase.util.MultiThreadedWriterBase

Direct Known Subclasses:

MultiThreadedUpdater, MultiThreadedWriter

public abstract class MultiThreadedWriterBase
extends MultiThreadedAction

Creates multiple threads that write key/values into the

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
private class	MultiThreadedWriterBase.WroteKeysTracker A thread that keeps track of the highest key in the contiguous range of inserted/updated keys.

Nested classes/interfaces inherited from class org.apache.hadoop.hbase.util.MultiThreadedAction

MultiThreadedAction.DefaultDataGenerator

Field Summary

Fields

Modifier and Type	Field and Description
protected Set<Long>	failedKeySet The sorted set of keys NOT inserted/updated by the writers
private static org.slf4j.Logger	LOG
protected AtomicLong	nextKeyToWrite This is the current key to be inserted/updated by any thread.
protected boolean	trackWroteKeys Enable this if used in conjunction with a concurrent reader.
protected AtomicLong	wroteKeyQueueSize The total size of the temporary inserted/updated key set that have not yet lined up in a our contiguous sequence starting from startKey.
protected BlockingQueue<Long>	wroteKeys A temporary place to keep track of inserted/updated keys.
protected AtomicLong	wroteUpToKey The highest key in the contiguous range of keys .

Fields inherited from class org.apache.hadoop.hbase.util.MultiThreadedAction

conf, connection, dataGenerator, endKey, numCols, numKeys, numThreads, numThreadsWorking, REPORTING_INTERVAL_MS, startKey, tableName, totalOpTimeMs, verbose

Constructor Summary

Constructors

Constructor and Description
MultiThreadedWriterBase(LoadTestDataGenerator dataGen, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.hbase.TableName tableName, String actionLetter)

Method Summary

Modifier and Type	Method and Description
protected BlockingQueue<Long>	createWriteKeysQueue(org.apache.hadoop.conf.Configuration conf)
boolean	failedToWriteKey(long k)
int	getNumWriteFailures()
protected String	getRegionDebugInfoSafe(org.apache.hadoop.hbase.client.Table table, byte[] rowKey)
protected String	progressInfo() Returns a task-specific progress string
void	setTrackWroteKeys(boolean enable) Used for a joint write/read workload.
void	start(long startKey, long endKey, int numThreads)
long	wroteUpToKey() The max key until which all keys have been inserted/updated (successfully or not).

Methods inherited from class org.apache.hadoop.hbase.util.MultiThreadedAction

appendToStatus, appendToStatus, close, getEndKey, isDone, startThreads, verifyResultAgainstDataGenerator, verifyResultAgainstDataGenerator, waitForFinish

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

private static final org.slf4j.Logger LOG

wroteKeys

protected BlockingQueue<Long> wroteKeys

A temporary place to keep track of inserted/updated keys. This is written to by all writers and is drained on a separate thread that populates wroteUpToKey, the maximum key in the contiguous range of keys being inserted/updated. This queue is supposed to stay small.

nextKeyToWrite

protected AtomicLong nextKeyToWrite

This is the current key to be inserted/updated by any thread. Each thread does an atomic get and increment operation and inserts the current value.

wroteUpToKey

protected AtomicLong wroteUpToKey

The highest key in the contiguous range of keys .

failedKeySet

protected Set<Long> failedKeySet

The sorted set of keys NOT inserted/updated by the writers

wroteKeyQueueSize

protected AtomicLong wroteKeyQueueSize

The total size of the temporary inserted/updated key set that have not yet lined up in a our contiguous sequence starting from startKey. Supposed to stay small.

trackWroteKeys

protected boolean trackWroteKeys

Enable this if used in conjunction with a concurrent reader.

Constructor Detail

MultiThreadedWriterBase

public MultiThreadedWriterBase(LoadTestDataGenerator dataGen,
                               org.apache.hadoop.conf.Configuration conf,
                               org.apache.hadoop.hbase.TableName tableName,
                               String actionLetter)
                        throws IOException

Throws:

IOException

Method Detail

createWriteKeysQueue

protected BlockingQueue<Long> createWriteKeysQueue(org.apache.hadoop.conf.Configuration conf)

start

public void start(long startKey,
                  long endKey,
                  int numThreads)
           throws IOException

Overrides:

start in class MultiThreadedAction

Throws:

IOException

getRegionDebugInfoSafe

protected String getRegionDebugInfoSafe(org.apache.hadoop.hbase.client.Table table,
                                        byte[] rowKey)

getNumWriteFailures

public int getNumWriteFailures()

wroteUpToKey

public long wroteUpToKey()

The max key until which all keys have been inserted/updated (successfully or not).

Returns:

the last key that we have inserted/updated all keys up to (inclusive)

failedToWriteKey

public boolean failedToWriteKey(long k)

progressInfo

protected String progressInfo()

Description copied from class: MultiThreadedAction

Returns a task-specific progress string

Specified by:

progressInfo in class MultiThreadedAction

setTrackWroteKeys

public void setTrackWroteKeys(boolean enable)

Used for a joint write/read workload. Enables tracking the last inserted/updated key, which requires a blocking queue and a consumer thread.

Parameters:

enable - whether to enable tracking the last inserted/updated key