Class MemStoreFlusher
java.lang.Object
org.apache.hadoop.hbase.regionserver.MemStoreFlusher
- All Implemented Interfaces:
ConfigurationObserver
,FlushRequester
@Private
public class MemStoreFlusher
extends Object
implements FlushRequester, ConfigurationObserver
Thread that flushes cache on request NOTE: This class extends Thread rather than Chore because
the sleep time can be interrupted when there is something to do, rather than the Chore sleep time
which is invariant.
- See Also:
-
Nested Class Summary
Modifier and TypeClassDescriptionprivate class
(package private) static interface
(package private) static class
Datastructure used in the flush queue. -
Field Summary
Modifier and TypeFieldDescriptionprivate long
private final Object
private org.apache.hadoop.conf.Configuration
private final AtomicInteger
private ThreadFactory
private MemStoreFlusher.FlushHandler[]
private final BlockingQueue<MemStoreFlusher.FlushQueueEntry>
private List<FlushRequestListener>
private final ReentrantReadWriteLock
private static final org.slf4j.Logger
protected final Map<Region,
MemStoreFlusher.FlushRegionEntry> private final HRegionServer
private final long
private final LongAdder
private static final MemStoreFlusher.FlushQueueEntry
Singleton instance inserted into flush queue used for signaling.private AtomicBoolean
-
Constructor Summary
ConstructorDescriptionMemStoreFlusher
(org.apache.hadoop.conf.Configuration conf, HRegionServer server) -
Method Summary
Modifier and TypeMethodDescriptionprivate boolean
flushOneForGlobalPressure
(FlushType flushType) The memstore across all regions has exceeded the low water mark.private boolean
flushRegion
(HRegion region, boolean emergencyFlush, List<byte[]> families, FlushLifeCycleTracker tracker) Flush a region.private boolean
A flushRegion that checks store file count.private HRegion
getBiggestMemStoreOfRegionReplica
(SortedMap<Long, Collection<HRegion>> regionsBySize, Set<HRegion> excludedRegions) private HRegion
getBiggestMemStoreRegion
(SortedMap<Long, Collection<HRegion>> regionsBySize, Set<HRegion> excludedRegions, boolean checkStoreFileCount) int
int
private int
getHandlerCount
(org.apache.hadoop.conf.Configuration conf) private static long
Returns Return memstore data size or null ifr
is nullprivate static long
Returns Return memstore heap size or null ifr
is nullprivate static long
Returns Return memstore offheap size or null ifr
is nullprivate int
getStoreFileCount
(Region region) (package private) void
Only interrupt once it's done with a run through the work loop.private FlushType
Return the FlushType if global memory usage is above the high watermarkprivate FlushType
Return the FlushType if we're above the low watermark(package private) boolean
isAlive()
private boolean
isTooManyStoreFiles
(Region region) private void
private void
notifyFlushRequest
(Region region, boolean emergencyFlush) void
onConfigurationChange
(org.apache.hadoop.conf.Configuration newConf) This method would be called by theConfigurationManager
object when theConfiguration
object is reloaded from disk.void
Check if the regionserver's memstore memory usage is greater than the limit.private boolean
void
Register a MemstoreFlushListenerboolean
requestDelayedFlush
(HRegion r, long delay) Tell the listener the cache needs to be flushed after a delayboolean
requestFlush
(HRegion r, List<byte[]> families, FlushLifeCycleTracker tracker) Tell the listener the cache needs to be flushed.boolean
requestFlush
(HRegion r, FlushLifeCycleTracker tracker) Tell the listener the cache needs to be flushed.void
setGlobalMemStoreLimit
(long globalMemStoreSize) Sets the global memstore limit to a new size.(package private) void
shutdown()
(package private) void
private void
startFlushHandlerThreads
(MemStoreFlusher.FlushHandler[] flushHandlers, int start, int end) private void
stopFlushHandlerThreads
(MemStoreFlusher.FlushHandler[] flushHandlers, int start, int end) toString()
boolean
Unregister the listener from MemstoreFlushListenersprivate void
private void
-
Field Details
-
LOG
-
conf
-
flushQueue
-
regionsInQueue
-
wakeupPending
-
threadWakeFrequency
-
server
-
lock
-
blockSignal
-
blockingWaitTime
-
updatesBlockedMsHighWater
-
flushHandlers
-
flusherIdGen
-
flusherThreadFactory
-
flushRequestListeners
-
WAKEUPFLUSH_INSTANCE
Singleton instance inserted into flush queue used for signaling.
-
-
Constructor Details
-
MemStoreFlusher
-
-
Method Details
-
getUpdatesBlockedMsHighWater
-
flushOneForGlobalPressure
The memstore across all regions has exceeded the low water mark. Pick one region to flush and flush it synchronously (this is called from the flush thread)- Returns:
- true if successful
-
getMemStoreOffHeapSize
Returns Return memstore offheap size or null ifr
is null -
getMemStoreHeapSize
Returns Return memstore heap size or null ifr
is null -
getMemStoreDataSize
Returns Return memstore data size or null ifr
is null -
wakeupFlushThread
-
getBiggestMemStoreRegion
private HRegion getBiggestMemStoreRegion(SortedMap<Long, Collection<HRegion>> regionsBySize, Set<HRegion> excludedRegions, boolean checkStoreFileCount) -
getBiggestMemStoreOfRegionReplica
private HRegion getBiggestMemStoreOfRegionReplica(SortedMap<Long, Collection<HRegion>> regionsBySize, Set<HRegion> excludedRegions) -
refreshStoreFilesAndReclaimMemory
-
isAboveHighWaterMark
Return the FlushType if global memory usage is above the high watermark -
isAboveLowWaterMark
Return the FlushType if we're above the low watermark -
requestFlush
Description copied from interface:FlushRequester
Tell the listener the cache needs to be flushed.- Specified by:
requestFlush
in interfaceFlushRequester
- Parameters:
r
- the Region requesting the cache flush- Returns:
- true if our region is added into the queue, false otherwise
-
requestFlush
Description copied from interface:FlushRequester
Tell the listener the cache needs to be flushed.- Specified by:
requestFlush
in interfaceFlushRequester
- Parameters:
r
- the Region requesting the cache flushfamilies
- stores of region to flush, if null then use flush policy- Returns:
- true if our region is added into the queue, false otherwise
-
requestDelayedFlush
Description copied from interface:FlushRequester
Tell the listener the cache needs to be flushed after a delay- Specified by:
requestDelayedFlush
in interfaceFlushRequester
- Parameters:
r
- the Region requesting the cache flushdelay
- after how much time should the flush happen- Returns:
- true if our region is added into the queue, false otherwise
-
getFlushQueueSize
-
interruptIfNecessary
void interruptIfNecessary()Only interrupt once it's done with a run through the work loop. -
start
-
isAlive
boolean isAlive() -
shutdown
void shutdown() -
flushRegion
A flushRegion that checks store file count. If too many, puts the flush on delay queue to retry later.- Returns:
- true if the region was successfully flushed, false otherwise. If false, there will be accompanying log messages explaining why the region was not flushed.
-
flushRegion
private boolean flushRegion(HRegion region, boolean emergencyFlush, List<byte[]> families, FlushLifeCycleTracker tracker) Flush a region.- Parameters:
region
- Region to flush.emergencyFlush
- Set if we are being force flushed. If true the region needs to be removed from the flush queue. If false, when we were called from the main flusher run loop and we got the entry to flush by calling poll on the flush queue (which removed it).families
- stores of region to flush.- Returns:
- true if the region was successfully flushed, false otherwise. If false, there will be accompanying log messages explaining why the region was not flushed.
-
notifyFlushRequest
-
wakeUpIfBlocking
-
isTooManyStoreFiles
-
getStoreFileCount
-
reclaimMemStoreMemory
Check if the regionserver's memstore memory usage is greater than the limit. If so, flush regions with the biggest memstores until we're down to the lower limit. This method blocks callers until we're down to a safe amount of memstore consumption. -
logMsg
-
toString
-
dumpQueue
-
registerFlushRequestListener
Register a MemstoreFlushListener- Specified by:
registerFlushRequestListener
in interfaceFlushRequester
-
unregisterFlushRequestListener
Unregister the listener from MemstoreFlushListeners- Specified by:
unregisterFlushRequestListener
in interfaceFlushRequester
- Returns:
- true when passed listener is unregistered successfully.
-
setGlobalMemStoreLimit
Sets the global memstore limit to a new size.- Specified by:
setGlobalMemStoreLimit
in interfaceFlushRequester
-
getHandlerCount
-
onConfigurationChange
Description copied from interface:ConfigurationObserver
This method would be called by theConfigurationManager
object when theConfiguration
object is reloaded from disk.- Specified by:
onConfigurationChange
in interfaceConfigurationObserver
-
startFlushHandlerThreads
private void startFlushHandlerThreads(MemStoreFlusher.FlushHandler[] flushHandlers, int start, int end) -
stopFlushHandlerThreads
private void stopFlushHandlerThreads(MemStoreFlusher.FlushHandler[] flushHandlers, int start, int end) -
getFlusherCount
-