Package org.apache.hadoop.hbase.io.hfile

Class MemcachedBlockCache

java.lang.Object

org.apache.hadoop.hbase.io.hfile.MemcachedBlockCache

All Implemented Interfaces:

Iterable<CachedBlock>, BlockCache

@Private
public class MemcachedBlockCache
extends Object
implements BlockCache

Class to store blocks into memcached. This should only be used on a cluster of Memcached daemons that are tuned well and have a good network connection to the HBase regionservers. Any other use will likely slow down HBase greatly.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
private static class	MemcachedBlockCache.HFileBlockTranscoder	Class to encode and decode an HFileBlock to and from memecached's resulting byte arrays.
private static class	MemcachedBlockCache.StatisticsThread

Field Summary

Fields

Modifier and Type	Field	Description
private final AtomicLong	cachedCount
private final AtomicLong	cacheErrorCount
private final CacheStats	cacheStats
private final net.spy.memcached.MemcachedClient	client
private static final org.slf4j.Logger	LOG
static final int	MAX_SIZE
static final int	MAX_TIME
static final String	MEMCACHED_CONFIG_KEY
static final long	MEMCACHED_DEFAULT_TIMEOUT
static final String	MEMCACHED_OPTIMEOUT_KEY
static final boolean	MEMCACHED_OPTIMIZE_DEFAULT
static final String	MEMCACHED_OPTIMIZE_KEY
static final String	MEMCACHED_TIMEOUT_KEY
private final AtomicLong	notCachedCount
private final ScheduledExecutorService	scheduleThreadPool	Statistics thread schedule pool (for heavy debugging, could remove)
static final int	STAT_THREAD_PERIOD
private final MemcachedBlockCache.HFileBlockTranscoder	tc
private final AtomicLong	timeoutCount

Constructor Summary

Constructors

Constructor	Description
MemcachedBlockCache(org.apache.hadoop.conf.Configuration c)

Method Summary

Modifier and Type	Method	Description
void	cacheBlock(BlockCacheKey cacheKey, Cacheable buf)	Add block to cache (defaults to not in-memory).
void	cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean inMemory)	Add block to cache.
protected net.spy.memcached.MemcachedClient	createMemcachedClient(net.spy.memcached.ConnectionFactory factory, List<InetSocketAddress> serverAddresses)
boolean	evictBlock(BlockCacheKey cacheKey)	Evict block from cache.
int	evictBlocksByHfileName(String hfileName)	This method does nothing so that memcached can handle all evictions.
Cacheable	getBlock(BlockCacheKey cacheKey, boolean caching, boolean repeat, boolean updateCacheMetrics)	Fetch block from cache.
BlockCache[]	getBlockCaches()	Returns The list of sub blockcaches that make up this one; returns null if no sub caches.
long	getBlockCount()	Returns the number of blocks currently cached in the block cache.
long	getCurrentDataSize()	Returns the occupied size of data blocks, in bytes.
long	getCurrentSize()	Returns the occupied size of the block cache, in bytes.
long	getDataBlockCount()	Returns the number of data blocks currently cached in the block cache.
long	getFreeSize()	Returns the free size of the block cache, in bytes.
long	getMaxSize()	Returns the Max size of the block cache, in bytes.
CacheStats	getStats()	Get the statistics for this block cache.
Iterator<CachedBlock>	iterator()	Returns Iterator over the blocks in the cache.
void	logStats()
void	shutdown()	Shutdown the cache.
long	size()	Returns the total size of the block cache, in bytes.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.hbase.io.hfile.BlockCache

blockFitsIntoTheCache, cacheBlock, evictBlocksRangeByHfileName, getBlock, getBlockSize, getFullyCachedFiles, getRegionCachedInfo, isAlreadyCached, isCacheEnabled, isMetaBlock, notifyFileCachingCompleted, shouldCacheFile, waitForCacheInitialization

Methods inherited from interface java.lang.Iterable

forEach, spliterator

Field Details

LOG

private static final org.slf4j.Logger LOG

MAX_SIZE

public static final int MAX_SIZE

See Also:

• Constant Field Values

MAX_TIME

public static final int MAX_TIME

See Also:

• Constant Field Values

MEMCACHED_CONFIG_KEY

public static final String MEMCACHED_CONFIG_KEY

See Also:

• Constant Field Values

MEMCACHED_TIMEOUT_KEY

public static final String MEMCACHED_TIMEOUT_KEY

See Also:

• Constant Field Values

MEMCACHED_OPTIMEOUT_KEY

public static final String MEMCACHED_OPTIMEOUT_KEY

See Also:

• Constant Field Values

MEMCACHED_OPTIMIZE_KEY

public static final String MEMCACHED_OPTIMIZE_KEY

See Also:

• Constant Field Values

MEMCACHED_DEFAULT_TIMEOUT

public static final long MEMCACHED_DEFAULT_TIMEOUT

See Also:

• Constant Field Values

MEMCACHED_OPTIMIZE_DEFAULT

public static final boolean MEMCACHED_OPTIMIZE_DEFAULT

See Also:

• Constant Field Values

STAT_THREAD_PERIOD

public static final int STAT_THREAD_PERIOD

See Also:

• Constant Field Values

client

private final net.spy.memcached.MemcachedClient client

tc

private final MemcachedBlockCache.HFileBlockTranscoder tc

cacheStats

private final CacheStats cacheStats

cachedCount

private final AtomicLong cachedCount

notCachedCount

private final AtomicLong notCachedCount

cacheErrorCount

private final AtomicLong cacheErrorCount

timeoutCount

private final AtomicLong timeoutCount

scheduleThreadPool

private final transient ScheduledExecutorService scheduleThreadPool

Statistics thread schedule pool (for heavy debugging, could remove)

Constructor Details

MemcachedBlockCache

public MemcachedBlockCache(org.apache.hadoop.conf.Configuration c)
                    throws IOException

Throws:

IOException

Method Details

createMemcachedClient

protected net.spy.memcached.MemcachedClient createMemcachedClient(net.spy.memcached.ConnectionFactory factory,
 List<InetSocketAddress> serverAddresses)
                                                           throws IOException

Throws:

IOException

cacheBlock

public void cacheBlock(BlockCacheKey cacheKey,
 Cacheable buf,
 boolean inMemory)

Description copied from interface: BlockCache

Add block to cache.

Specified by:

cacheBlock in interface BlockCache

Parameters:

cacheKey - The block's cache key.

buf - The block contents wrapped in a ByteBuffer.

inMemory - Whether block should be treated as in-memory

cacheBlock

public void cacheBlock(BlockCacheKey cacheKey,
 Cacheable buf)

Description copied from interface: BlockCache

Add block to cache (defaults to not in-memory).

Specified by:

cacheBlock in interface BlockCache

Parameters:

cacheKey - The block's cache key.

buf - The object to cache.

getBlock

public Cacheable getBlock(BlockCacheKey cacheKey,
 boolean caching,
 boolean repeat,
 boolean updateCacheMetrics)

Description copied from interface: BlockCache

Fetch block from cache.

Specified by:

getBlock in interface BlockCache

Parameters:

cacheKey - Block to fetch.

caching - Whether this request has caching enabled (used for stats)

repeat - Whether this is a repeat lookup for the same block (used to avoid double counting cache misses when doing double-check locking)

updateCacheMetrics - Whether to update cache metrics or not

Returns:

Block or null if block is not in 2 cache.

evictBlock

public boolean evictBlock(BlockCacheKey cacheKey)

Description copied from interface: BlockCache

Evict block from cache.

Specified by:

evictBlock in interface BlockCache

Parameters:

cacheKey - Block to evict

Returns:

true if block existed and was evicted, false if not

evictBlocksByHfileName

public int evictBlocksByHfileName(String hfileName)

This method does nothing so that memcached can handle all evictions.

Specified by:

evictBlocksByHfileName in interface BlockCache

Returns:

the number of blocks evicted

getStats

public CacheStats getStats()

Description copied from interface: BlockCache

Get the statistics for this block cache.

Specified by:

getStats in interface BlockCache

shutdown

public void shutdown()

Description copied from interface: BlockCache

Shutdown the cache.

Specified by:

shutdown in interface BlockCache

size

public long size()

Description copied from interface: BlockCache

Returns the total size of the block cache, in bytes.

Specified by:

size in interface BlockCache

Returns:

size of cache, in bytes

getMaxSize

public long getMaxSize()

Description copied from interface: BlockCache

Returns the Max size of the block cache, in bytes.

Specified by:

getMaxSize in interface BlockCache

Returns:

size of cache, in bytes

getFreeSize

public long getFreeSize()

Description copied from interface: BlockCache

Returns the free size of the block cache, in bytes.

Specified by:

getFreeSize in interface BlockCache

Returns:

free space in cache, in bytes

getCurrentSize

public long getCurrentSize()

Description copied from interface: BlockCache

Returns the occupied size of the block cache, in bytes.

Specified by:

getCurrentSize in interface BlockCache

Returns:

occupied space in cache, in bytes

getCurrentDataSize

public long getCurrentDataSize()

Description copied from interface: BlockCache

Returns the occupied size of data blocks, in bytes.

Specified by:

getCurrentDataSize in interface BlockCache

Returns:

occupied space in cache, in bytes

getBlockCount

public long getBlockCount()

Description copied from interface: BlockCache

Returns the number of blocks currently cached in the block cache.

Specified by:

getBlockCount in interface BlockCache

Returns:

number of blocks in the cache

getDataBlockCount

public long getDataBlockCount()

Description copied from interface: BlockCache

Returns the number of data blocks currently cached in the block cache.

Specified by:

getDataBlockCount in interface BlockCache

Returns:

number of blocks in the cache

iterator

public Iterator<CachedBlock> iterator()

Description copied from interface: BlockCache

Returns Iterator over the blocks in the cache.

Specified by:

iterator in interface BlockCache

Specified by:

iterator in interface Iterable<CachedBlock>

getBlockCaches

public BlockCache[] getBlockCaches()

Description copied from interface: BlockCache

Returns The list of sub blockcaches that make up this one; returns null if no sub caches.

Specified by:

getBlockCaches in interface BlockCache

logStats

public void logStats()