org.apache.hadoop.hbase.regionserver

Class InputStreamBlockDistribution

java.lang.Object

org.apache.hadoop.hbase.regionserver.InputStreamBlockDistribution

@InterfaceAudience.Private
public class InputStreamBlockDistribution
extends Object

Computes the HDFSBlockDistribution for a file based on the underlying located blocks for an HdfsDataInputStream reading that file. The backing DFSInputStream.getAllBlocks involves allocating an array of numBlocks size per call. It may also involve calling the namenode, if the DFSInputStream has not fetched all the blocks yet. In order to avoid allocation pressure, we cache the computed distribution for a configurable period of time.

This class only gets instantiated for the first FSDataInputStream of each StoreFile (i.e. the one backing HStoreFile.initialReader). It's then used to dynamically update the value returned by HStoreFile.getHDFSBlockDistribution().

Once the backing FSDataInputStream is closed, we should not expect the distribution result to change anymore. This is ok becuase the initialReader's InputStream is only closed when the StoreFile itself is closed, at which point nothing will be querying getHDFSBlockDistribution anymore. If/When the StoreFile is reopened, a new InputStreamBlockDistribution will be created for the new initialReader.

Field Summary

Fields

Modifier and Type	Field and Description
private int	cachePeriodMs
private static int	DEFAULT_HBASE_LOCALITY_INPUTSTREAM_DERIVE_CACHE_PERIOD
private static boolean	DEFAULT_HBASE_LOCALITY_INPUTSTREAM_DERIVE_ENABLED
private StoreFileInfo	fileInfo
private static String	HBASE_LOCALITY_INPUTSTREAM_DERIVE_CACHE_PERIOD
private static String	HBASE_LOCALITY_INPUTSTREAM_DERIVE_ENABLED
private HDFSBlocksDistribution	hdfsBlocksDistribution
private long	lastCachedAt
private static org.slf4j.Logger	LOG
private org.apache.hadoop.fs.FSDataInputStream	stream
private boolean	streamUnsupported

Constructor Summary

Constructors

Constructor and Description
InputStreamBlockDistribution(org.apache.hadoop.fs.FSDataInputStream stream, StoreFileInfo fileInfo) This should only be called for the first FSDataInputStream of a StoreFile, in HStoreFile.open().

Method Summary

Modifier and Type	Method and Description
private void	computeBlockDistribution()
(package private) long	getCachePeriodMs() For tests only, returns the configured cache period
HDFSBlocksDistribution	getHDFSBlockDistribution() Get the HDFSBlocksDistribution derived from the StoreFile input stream, re-computing if cache is expired.
static boolean	isEnabled(org.apache.hadoop.conf.Configuration conf) True if we should derive StoreFile HDFSBlockDistribution from the underlying input stream
(package private) boolean	isStreamUnsupported() For tests only, returns whether the passed stream is supported
(package private) void	setLastCachedAt(long timestamp) For tests only, sets lastCachedAt so we can force a refresh

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

private static final org.slf4j.Logger LOG

HBASE_LOCALITY_INPUTSTREAM_DERIVE_ENABLED

private static final String HBASE_LOCALITY_INPUTSTREAM_DERIVE_ENABLED

See Also:

Constant Field Values

DEFAULT_HBASE_LOCALITY_INPUTSTREAM_DERIVE_ENABLED

private static final boolean DEFAULT_HBASE_LOCALITY_INPUTSTREAM_DERIVE_ENABLED

See Also:

Constant Field Values

HBASE_LOCALITY_INPUTSTREAM_DERIVE_CACHE_PERIOD

private static final String HBASE_LOCALITY_INPUTSTREAM_DERIVE_CACHE_PERIOD

See Also:

Constant Field Values

DEFAULT_HBASE_LOCALITY_INPUTSTREAM_DERIVE_CACHE_PERIOD

private static final int DEFAULT_HBASE_LOCALITY_INPUTSTREAM_DERIVE_CACHE_PERIOD

See Also:

Constant Field Values

stream

private final org.apache.hadoop.fs.FSDataInputStream stream

fileInfo

private final StoreFileInfo fileInfo

cachePeriodMs

private final int cachePeriodMs

hdfsBlocksDistribution

private HDFSBlocksDistribution hdfsBlocksDistribution

lastCachedAt

private long lastCachedAt

streamUnsupported

private boolean streamUnsupported

Constructor Detail

InputStreamBlockDistribution

public InputStreamBlockDistribution(org.apache.hadoop.fs.FSDataInputStream stream,
                                    StoreFileInfo fileInfo)

This should only be called for the first FSDataInputStream of a StoreFile, in HStoreFile.open().

Parameters:

stream - the input stream to derive locality from

fileInfo - the StoreFileInfo for the related store file

See Also:

InputStreamBlockDistribution

Method Detail

isEnabled

public static boolean isEnabled(org.apache.hadoop.conf.Configuration conf)

True if we should derive StoreFile HDFSBlockDistribution from the underlying input stream

getHDFSBlockDistribution

public HDFSBlocksDistribution getHDFSBlockDistribution()

Get the HDFSBlocksDistribution derived from the StoreFile input stream, re-computing if cache is expired.

computeBlockDistribution

private void computeBlockDistribution()
                               throws IOException

Throws:

IOException

setLastCachedAt

void setLastCachedAt(long timestamp)

For tests only, sets lastCachedAt so we can force a refresh

getCachePeriodMs

long getCachePeriodMs()

For tests only, returns the configured cache period

isStreamUnsupported

boolean isStreamUnsupported()

For tests only, returns whether the passed stream is supported