org.apache.hadoop.hbase.regionserver

Class MemStoreFlusher

java.lang.Object

org.apache.hadoop.hbase.regionserver.MemStoreFlusher

All Implemented Interfaces:

FlushRequester

@InterfaceAudience.Private
public class MemStoreFlusher
extends Object
implements FlushRequester

Thread that flushes cache on request NOTE: This class extends Thread rather than Chore because the sleep time can be interrupted when there is something to do, rather than the Chore sleep time which is invariant.

See Also:

FlushRequester

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
private class	MemStoreFlusher.FlushHandler
(package private) static interface	MemStoreFlusher.FlushQueueEntry
(package private) static class	MemStoreFlusher.FlushRegionEntry Datastructure used in the flush queue.

Field Summary

Fields

Modifier and Type	Field and Description
private long	blockingWaitTime
private Object	blockSignal
private org.apache.hadoop.conf.Configuration	conf
private MemStoreFlusher.FlushHandler[]	flushHandlers
private BlockingQueue<MemStoreFlusher.FlushQueueEntry>	flushQueue
private List<FlushRequestListener>	flushRequestListeners
private ReentrantReadWriteLock	lock
private static org.slf4j.Logger	LOG
protected Map<Region,MemStoreFlusher.FlushRegionEntry>	regionsInQueue
private HRegionServer	server
private long	threadWakeFrequency
private LongAdder	updatesBlockedMsHighWater
private static MemStoreFlusher.FlushQueueEntry	WAKEUPFLUSH_INSTANCE Singleton instance inserted into flush queue used for signaling.
private AtomicBoolean	wakeupPending

Constructor Summary

Constructors

Constructor and Description
MemStoreFlusher(org.apache.hadoop.conf.Configuration conf, HRegionServer server) nn

Method Summary

Modifier and Type	Method and Description
String	dumpQueue()
private boolean	flushOneForGlobalPressure(FlushType flushType) The memstore across all regions has exceeded the low water mark.
private boolean	flushRegion(HRegion region, boolean emergencyFlush, List<byte[]> families, FlushLifeCycleTracker tracker) Flush a region.
private boolean	flushRegion(MemStoreFlusher.FlushRegionEntry fqe) A flushRegion that checks store file count.
private HRegion	getBiggestMemStoreOfRegionReplica(SortedMap<Long,HRegion> regionsBySize, Set<HRegion> excludedRegions)
private HRegion	getBiggestMemStoreRegion(SortedMap<Long,HRegion> regionsBySize, Set<HRegion> excludedRegions, boolean checkStoreFileCount)
int	getFlushQueueSize()
private static long	getMemStoreDataSize(HRegion r) Returns Return memstore data size or null if r is null
private static long	getMemStoreHeapSize(HRegion r) Returns Return memstore heap size or null if r is null
private static long	getMemStoreOffHeapSize(HRegion r) Returns Return memstore offheap size or null if r is null
private int	getStoreFileCount(Region region)
LongAdder	getUpdatesBlockedMsHighWater()
(package private) void	interruptIfNecessary() Only interrupt once it's done with a run through the work loop.
private FlushType	isAboveHighWaterMark() Return the FlushType if global memory usage is above the high watermark
private FlushType	isAboveLowWaterMark() Return the FlushType if we're above the low watermark
(package private) boolean	isAlive()
private boolean	isTooManyStoreFiles(Region region)
(package private) void	join()
private void	logMsg(String type, long val, long max)
private void	notifyFlushRequest(Region region, boolean emergencyFlush)
void	reclaimMemStoreMemory() Check if the regionserver's memstore memory usage is greater than the limit.
private boolean	refreshStoreFilesAndReclaimMemory(Region region)
void	registerFlushRequestListener(FlushRequestListener listener) Register a MemstoreFlushListener n
boolean	requestDelayedFlush(HRegion r, long delay) Tell the listener the cache needs to be flushed after a delay
boolean	requestFlush(HRegion r, FlushLifeCycleTracker tracker) Tell the listener the cache needs to be flushed.
boolean	requestFlush(HRegion r, List<byte[]> families, FlushLifeCycleTracker tracker) Tell the listener the cache needs to be flushed.
void	setGlobalMemStoreLimit(long globalMemStoreSize) Sets the global memstore limit to a new size.
(package private) void	start(Thread.UncaughtExceptionHandler eh)
String	toString()
boolean	unregisterFlushRequestListener(FlushRequestListener listener) Unregister the listener from MemstoreFlushListeners n * @return true when passed listener is unregistered successfully.
private void	wakeupFlushThread()
private void	wakeUpIfBlocking()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

LOG

private static final org.slf4j.Logger LOG

conf

private org.apache.hadoop.conf.Configuration conf

flushQueue

private final BlockingQueue<MemStoreFlusher.FlushQueueEntry> flushQueue

regionsInQueue

protected final Map<Region,MemStoreFlusher.FlushRegionEntry> regionsInQueue

wakeupPending

private AtomicBoolean wakeupPending

threadWakeFrequency

private final long threadWakeFrequency

server

private final HRegionServer server

lock

private final ReentrantReadWriteLock lock

blockSignal

private final Object blockSignal

blockingWaitTime

private long blockingWaitTime

updatesBlockedMsHighWater

private final LongAdder updatesBlockedMsHighWater

flushHandlers

private final MemStoreFlusher.FlushHandler[] flushHandlers

flushRequestListeners

private List<FlushRequestListener> flushRequestListeners

WAKEUPFLUSH_INSTANCE

private static final MemStoreFlusher.FlushQueueEntry WAKEUPFLUSH_INSTANCE

Singleton instance inserted into flush queue used for signaling.

Constructor Detail

MemStoreFlusher

public MemStoreFlusher(org.apache.hadoop.conf.Configuration conf,
                       HRegionServer server)

nn

Method Detail

getUpdatesBlockedMsHighWater

public LongAdder getUpdatesBlockedMsHighWater()

flushOneForGlobalPressure

private boolean flushOneForGlobalPressure(FlushType flushType)

The memstore across all regions has exceeded the low water mark. Pick one region to flush and flush it synchronously (this is called from the flush thread)

Returns:

true if successful

getMemStoreOffHeapSize

private static long getMemStoreOffHeapSize(HRegion r)

Returns Return memstore offheap size or null if r is null

getMemStoreHeapSize

private static long getMemStoreHeapSize(HRegion r)

Returns Return memstore heap size or null if r is null

getMemStoreDataSize

private static long getMemStoreDataSize(HRegion r)

Returns Return memstore data size or null if r is null

wakeupFlushThread

private void wakeupFlushThread()

getBiggestMemStoreRegion

private HRegion getBiggestMemStoreRegion(SortedMap<Long,HRegion> regionsBySize,
                                         Set<HRegion> excludedRegions,
                                         boolean checkStoreFileCount)

getBiggestMemStoreOfRegionReplica

private HRegion getBiggestMemStoreOfRegionReplica(SortedMap<Long,HRegion> regionsBySize,
                                                  Set<HRegion> excludedRegions)

refreshStoreFilesAndReclaimMemory

private boolean refreshStoreFilesAndReclaimMemory(Region region)

isAboveHighWaterMark

private FlushType isAboveHighWaterMark()

Return the FlushType if global memory usage is above the high watermark

isAboveLowWaterMark

private FlushType isAboveLowWaterMark()

Return the FlushType if we're above the low watermark

requestFlush

public boolean requestFlush(HRegion r,
                            FlushLifeCycleTracker tracker)

Description copied from interface: FlushRequester

Tell the listener the cache needs to be flushed.

Specified by:

requestFlush in interface FlushRequester

Parameters:

r - the Region requesting the cache flush

Returns:

true if our region is added into the queue, false otherwise

requestFlush

public boolean requestFlush(HRegion r,
                            List<byte[]> families,
                            FlushLifeCycleTracker tracker)

Description copied from interface: FlushRequester

Tell the listener the cache needs to be flushed.

Specified by:

requestFlush in interface FlushRequester

Parameters:

r - the Region requesting the cache flush

families - stores of region to flush, if null then use flush policy

Returns:

true if our region is added into the queue, false otherwise

requestDelayedFlush

public boolean requestDelayedFlush(HRegion r,
                                   long delay)

Description copied from interface: FlushRequester

Tell the listener the cache needs to be flushed after a delay

Specified by:

requestDelayedFlush in interface FlushRequester

Parameters:

r - the Region requesting the cache flush

delay - after how much time should the flush happen

Returns:

true if our region is added into the queue, false otherwise

getFlushQueueSize

public int getFlushQueueSize()

interruptIfNecessary

void interruptIfNecessary()

Only interrupt once it's done with a run through the work loop.

start

void start(Thread.UncaughtExceptionHandler eh)

isAlive

boolean isAlive()

join

void join()

flushRegion

private boolean flushRegion(MemStoreFlusher.FlushRegionEntry fqe)

A flushRegion that checks store file count. If too many, puts the flush on delay queue to retry later. n * @return true if the region was successfully flushed, false otherwise. If false, there will be accompanying log messages explaining why the region was not flushed.

flushRegion

private boolean flushRegion(HRegion region,
                            boolean emergencyFlush,
                            List<byte[]> families,
                            FlushLifeCycleTracker tracker)

Flush a region.

Parameters:

region - Region to flush.

emergencyFlush - Set if we are being force flushed. If true the region needs to be removed from the flush queue. If false, when we were called from the main flusher run loop and we got the entry to flush by calling poll on the flush queue (which removed it).

families - stores of region to flush.

Returns:

true if the region was successfully flushed, false otherwise. If false, there will be accompanying log messages explaining why the region was not flushed.

notifyFlushRequest

private void notifyFlushRequest(Region region,
                                boolean emergencyFlush)

wakeUpIfBlocking

private void wakeUpIfBlocking()

isTooManyStoreFiles

private boolean isTooManyStoreFiles(Region region)

getStoreFileCount

private int getStoreFileCount(Region region)

reclaimMemStoreMemory

public void reclaimMemStoreMemory()

Check if the regionserver's memstore memory usage is greater than the limit. If so, flush regions with the biggest memstores until we're down to the lower limit. This method blocks callers until we're down to a safe amount of memstore consumption.

logMsg

private void logMsg(String type,
                    long val,
                    long max)

toString

public String toString()

Overrides:

toString in class Object

dumpQueue

public String dumpQueue()

registerFlushRequestListener

public void registerFlushRequestListener(FlushRequestListener listener)

Register a MemstoreFlushListener n

Specified by:

registerFlushRequestListener in interface FlushRequester

unregisterFlushRequestListener

public boolean unregisterFlushRequestListener(FlushRequestListener listener)

Unregister the listener from MemstoreFlushListeners n * @return true when passed listener is unregistered successfully.

Specified by:

unregisterFlushRequestListener in interface FlushRequester

setGlobalMemStoreLimit

public void setGlobalMemStoreLimit(long globalMemStoreSize)

Sets the global memstore limit to a new size. n

Specified by:

setGlobalMemStoreLimit in interface FlushRequester