org.apache.hadoop.hbase.mapreduce

Class RoundRobinTableInputFormat

java.lang.Object

org.apache.hadoop.mapreduce.InputFormat<ImmutableBytesWritable,Result>

org.apache.hadoop.hbase.mapreduce.TableInputFormatBase

org.apache.hadoop.hbase.mapreduce.TableInputFormat

org.apache.hadoop.hbase.mapreduce.RoundRobinTableInputFormat

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable

@InterfaceAudience.Public
public class RoundRobinTableInputFormat
extends TableInputFormat

Process the return from super-class TableInputFormat (TIF) so as to undo any clumping of InputSplits around RegionServers. Spread splits broadly to distribute read-load over RegionServers in the cluster. The super-class TIF returns splits in hbase:meta table order. Adjacent or near-adjacent hbase:meta Regions can be hosted on the same RegionServer -- nothing prevents this. This hbase:maeta ordering of InputSplit placement can be lumpy making it so some RegionServers end up hosting lots of InputSplit scans while contemporaneously other RegionServers host few or none. This class does a pass over the return from the super-class to better spread the load. See the below helpful Flipkart blog post for a description and from where the base of this code comes from (with permission).

See Also:

https://tech.flipkart.com/is-data-locality-always-out-of-the-box-in-hadoop-not-really-2ae9c95163cb

Field Summary

Fields inherited from class org.apache.hadoop.hbase.mapreduce.TableInputFormat

INPUT_TABLE, SCAN, SCAN_BATCHSIZE, SCAN_CACHEBLOCKS, SCAN_CACHEDROWS, SCAN_COLUMN_FAMILY, SCAN_COLUMNS, SCAN_MAXVERSIONS, SCAN_ROW_START, SCAN_ROW_STOP, SCAN_TIMERANGE_END, SCAN_TIMERANGE_START, SCAN_TIMESTAMP, SHUFFLE_MAPS

Fields inherited from class org.apache.hadoop.hbase.mapreduce.TableInputFormatBase

MAPREDUCE_INPUT_AUTOBALANCE, MAX_AVERAGE_REGION_SIZE, NUM_MAPPERS_PER_REGION

Constructor Summary

Constructors

Constructor and Description
RoundRobinTableInputFormat()

Method Summary

Modifier and Type	Method and Description
List<org.apache.hadoop.mapreduce.InputSplit>	getSplits(org.apache.hadoop.mapreduce.JobContext context) Calculates the splits that will serve as input for the map tasks.
static void	main(String[] args) Pass table name as argument.

Methods inherited from class org.apache.hadoop.hbase.mapreduce.TableInputFormat

addColumns, configureSplitTable, createScanFromConfiguration, getConf, getStartEndKeys, initialize, setConf

Methods inherited from class org.apache.hadoop.hbase.mapreduce.TableInputFormatBase

calculateAutoBalancedSplits, closeTable, createNInputSplitsUniform, createRecordReader, getAdmin, getRegionLocator, getScan, getTable, includeRegionInSplit, initializeTable, setScan, setTableRecordReader

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

RoundRobinTableInputFormat

public RoundRobinTableInputFormat()

Method Detail

getSplits

public List<org.apache.hadoop.mapreduce.InputSplit> getSplits(org.apache.hadoop.mapreduce.JobContext context)
                                                       throws IOException

Description copied from class: TableInputFormat

Calculates the splits that will serve as input for the map tasks. The number of splits matches the number of regions in a table. Splits are shuffled if required.

Overrides:

getSplits in class TableInputFormat

Parameters:

context - The current job context.

Returns:

The list of input splits.

Throws:

IOException - When creating the list of splits fails.

See Also:

InputFormat.getSplits( org.apache.hadoop.mapreduce.JobContext)

main

public static void main(String[] args)
                 throws IOException

Pass table name as argument. Set the zk ensemble to use with the System property 'hbase.zookeeper.quorum'

Throws:

IOException