org.apache.hadoop.hbase.master.normalizer

Class SimpleRegionNormalizer

java.lang.Object

org.apache.hadoop.hbase.master.normalizer.SimpleRegionNormalizer

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, RegionNormalizer

@InterfaceAudience.LimitedPrivate(value="Configuration")
public class SimpleRegionNormalizer
extends Object
implements RegionNormalizer

Simple implementation of region normalizer. Logic in use:

1. Get all regions of a given table
2. Get avg size S of the regions in the table (by total size of store files reported in RegionMetrics)
3. For each region R0, if R0 is bigger than S * 2, it is kindly requested to split.
4. Otherwise, for the next region in the chain R1, if R0 + R1 is smaller then S, R0 and R1 are kindly requested to merge.

The following parameters are configurable:

1. Whether to split a region as part of normalization. Configuration: "hbase.normalizer.split.enabled", default: true.
2. Whether to merge a region as part of normalization. Configuration: "hbase.normalizer.merge.enabled", default: true.
3. The minimum number of regions in a table to consider it for merge normalization. Configuration: "hbase.normalizer.min.region.count", default: 3.
4. The minimum age for a region to be considered for a merge, in days. Configuration: "hbase.normalizer.merge.min_region_age.days", default: 3.
5. The minimum size for a region to be considered for a merge, in whole MBs. Configuration: "hbase.normalizer.merge.min_region_size.mb", default: 1.

To see detailed logging of the application of these configuration values, set the log level for this class to `TRACE`.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
private class	SimpleRegionNormalizer.NormalizeContext Inner class caries the state necessary to perform a single invocation of computePlansForTable(TableName).

Field Summary

Fields

Modifier and Type	Field and Description
private org.apache.hadoop.conf.Configuration	conf
(package private) static boolean	DEFAULT_MERGE_ENABLED
(package private) static int	DEFAULT_MERGE_MIN_REGION_AGE_DAYS
(package private) static int	DEFAULT_MERGE_MIN_REGION_SIZE_MB
(package private) static int	DEFAULT_MIN_REGION_COUNT
(package private) static boolean	DEFAULT_SPLIT_ENABLED
private static org.slf4j.Logger	LOG
private MasterServices	masterServices
(package private) static String	MERGE_ENABLED_KEY
(package private) static String	MERGE_MIN_REGION_AGE_DAYS_KEY
(package private) static String	MERGE_MIN_REGION_SIZE_MB_KEY
private boolean	mergeEnabled
private Period	mergeMinRegionAge
private int	mergeMinRegionSizeMb
(package private) static String	MIN_REGION_COUNT_KEY
private int	minRegionCount
private long[]	skippedCount
(package private) static String	SPLIT_ENABLED_KEY
private boolean	splitEnabled

Constructor Summary

Constructors

Constructor and Description
SimpleRegionNormalizer()

Method Summary

Modifier and Type	Method and Description
private List<NormalizationPlan>	computeMergeNormalizationPlans(SimpleRegionNormalizer.NormalizeContext ctx) Computes the merge plans that should be executed for this table to converge average region towards target average or target region count.
List<NormalizationPlan>	computePlansForTable(TableName table) Computes a list of normalizer actions to perform on the target table.
private List<NormalizationPlan>	computeSplitNormalizationPlans(SimpleRegionNormalizer.NormalizeContext ctx) Computes the split plans that should be executed for this table to converge average region size towards target average or target region count.
private double	getAverageRegionSizeMb(List<RegionInfo> tableRegions)
org.apache.hadoop.conf.Configuration	getConf()
Period	getMergeMinRegionAge() Return this instance's configured value for "hbase.normalizer.merge.min_region_age.days".
int	getMergeMinRegionSizeMb() Return this instance's configured value for "hbase.normalizer.merge.min_region_size.mb".
int	getMinRegionCount() Return this instance's configured value for "hbase.normalizer.min.region.count".
private long	getRegionSizeMB(RegionInfo hri)
long	getSkippedCount(NormalizationPlan.PlanType type)
private boolean	isLargeEnoughForMerge(RegionInfo regionInfo) Return true when regionInfo has a size that is sufficient to be considered for a merge operation, false otherwise.
private boolean	isMasterSwitchEnabled(MasterSwitchType masterSwitchType)
boolean	isMergeEnabled() Return this instance's configured value for "hbase.normalizer.merge.enabled".
private boolean	isOldEnoughForMerge(RegionInfo regionInfo) Return true when regionInfo has a creation date that is old enough to be considered for a merge operation, false otherwise.
boolean	isSplitEnabled() Return this instance's configured value for "hbase.normalizer.split.enabled".
private static boolean	logTraceReason(BooleanSupplier predicate, String fmtWhenTrue, Object... args)
private static Period	parseMergeMinRegionAge(org.apache.hadoop.conf.Configuration conf)
private static int	parseMergeMinRegionSizeMb(org.apache.hadoop.conf.Configuration conf)
private static int	parseMinRegionCount(org.apache.hadoop.conf.Configuration conf)
void	planSkipped(RegionInfo hri, NormalizationPlan.PlanType type) Notification for the case where plan couldn't be executed due to constraint violation, such as namespace quota
private boolean	proceedWithMergePlanning()
private boolean	proceedWithSplitPlanning()
void	setConf(org.apache.hadoop.conf.Configuration conf)
void	setMasterServices(MasterServices masterServices) Set the master service.
private boolean	skipForMerge(RegionStates regionStates, RegionInfo regionInfo) Determine if a RegionInfo should be considered for a merge operation.
private static boolean	skipForSplit(RegionState state, RegionInfo regionInfo) Determine if a region in RegionState should be considered for a split operation.
private static <T> void	warnInvalidValue(String key, T parsedValue, T settledValue)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

private static final org.slf4j.Logger LOG

SPLIT_ENABLED_KEY

static final String SPLIT_ENABLED_KEY

See Also:

Constant Field Values

DEFAULT_SPLIT_ENABLED

static final boolean DEFAULT_SPLIT_ENABLED

See Also:

Constant Field Values

MERGE_ENABLED_KEY

static final String MERGE_ENABLED_KEY

See Also:

Constant Field Values

DEFAULT_MERGE_ENABLED

static final boolean DEFAULT_MERGE_ENABLED

See Also:

Constant Field Values

MIN_REGION_COUNT_KEY

static final String MIN_REGION_COUNT_KEY

See Also:

Constant Field Values

DEFAULT_MIN_REGION_COUNT

static final int DEFAULT_MIN_REGION_COUNT

See Also:

Constant Field Values

MERGE_MIN_REGION_AGE_DAYS_KEY

static final String MERGE_MIN_REGION_AGE_DAYS_KEY

See Also:

Constant Field Values

DEFAULT_MERGE_MIN_REGION_AGE_DAYS

static final int DEFAULT_MERGE_MIN_REGION_AGE_DAYS

See Also:

Constant Field Values

MERGE_MIN_REGION_SIZE_MB_KEY

static final String MERGE_MIN_REGION_SIZE_MB_KEY

See Also:

Constant Field Values

DEFAULT_MERGE_MIN_REGION_SIZE_MB

static final int DEFAULT_MERGE_MIN_REGION_SIZE_MB

See Also:

Constant Field Values

skippedCount

private final long[] skippedCount

conf

private org.apache.hadoop.conf.Configuration conf

masterServices

private MasterServices masterServices

splitEnabled

private boolean splitEnabled

mergeEnabled

private boolean mergeEnabled

minRegionCount

private int minRegionCount

mergeMinRegionAge

private Period mergeMinRegionAge

mergeMinRegionSizeMb

private int mergeMinRegionSizeMb

Constructor Detail

SimpleRegionNormalizer

public SimpleRegionNormalizer()

Method Detail

getConf

public org.apache.hadoop.conf.Configuration getConf()

Specified by:

getConf in interface org.apache.hadoop.conf.Configurable

setConf

public void setConf(org.apache.hadoop.conf.Configuration conf)

Specified by:

setConf in interface org.apache.hadoop.conf.Configurable

parseMinRegionCount

private static int parseMinRegionCount(org.apache.hadoop.conf.Configuration conf)

parseMergeMinRegionAge

private static Period parseMergeMinRegionAge(org.apache.hadoop.conf.Configuration conf)

parseMergeMinRegionSizeMb

private static int parseMergeMinRegionSizeMb(org.apache.hadoop.conf.Configuration conf)

warnInvalidValue

private static <T> void warnInvalidValue(String key,
                                         T parsedValue,
                                         T settledValue)

isSplitEnabled

public boolean isSplitEnabled()

Return this instance's configured value for "hbase.normalizer.split.enabled".

isMergeEnabled

public boolean isMergeEnabled()

Return this instance's configured value for "hbase.normalizer.merge.enabled".

getMinRegionCount

public int getMinRegionCount()

Return this instance's configured value for "hbase.normalizer.min.region.count".

getMergeMinRegionAge

public Period getMergeMinRegionAge()

Return this instance's configured value for "hbase.normalizer.merge.min_region_age.days".

getMergeMinRegionSizeMb

public int getMergeMinRegionSizeMb()

Return this instance's configured value for "hbase.normalizer.merge.min_region_size.mb".

setMasterServices

public void setMasterServices(MasterServices masterServices)

Description copied from interface: RegionNormalizer

Set the master service. Must be called before first call to RegionNormalizer.computePlansForTable(TableName).

Specified by:

setMasterServices in interface RegionNormalizer

Parameters:

masterServices - master services to use

planSkipped

public void planSkipped(RegionInfo hri,
                        NormalizationPlan.PlanType type)

Description copied from interface: RegionNormalizer

Notification for the case where plan couldn't be executed due to constraint violation, such as namespace quota

Specified by:

planSkipped in interface RegionNormalizer

Parameters:

hri - the region which is involved in the plan

type - type of plan

getSkippedCount

public long getSkippedCount(NormalizationPlan.PlanType type)

Specified by:

getSkippedCount in interface RegionNormalizer

Parameters:

type - type of plan for which skipped count is to be returned

Returns:

the count of plans of specified type which were skipped

computePlansForTable

public List<NormalizationPlan> computePlansForTable(TableName table)

Description copied from interface: RegionNormalizer

Computes a list of normalizer actions to perform on the target table. This is the primary entry-point from the Master driving a normalization activity.

Specified by:

computePlansForTable in interface RegionNormalizer

Parameters:

table - table to normalize

Returns:

A list of the normalization actions to perform, or an empty list if there's nothing to do.

getRegionSizeMB

private long getRegionSizeMB(RegionInfo hri)

Returns:

size of region in MB and if region is not found than -1

isMasterSwitchEnabled

private boolean isMasterSwitchEnabled(MasterSwitchType masterSwitchType)

proceedWithSplitPlanning

private boolean proceedWithSplitPlanning()

proceedWithMergePlanning

private boolean proceedWithMergePlanning()

getAverageRegionSizeMb

private double getAverageRegionSizeMb(List<RegionInfo> tableRegions)

Parameters:

tableRegions - regions of table to normalize

Returns:

average region size depending on

See Also:

Also make sure tableRegions contains regions of the same table

skipForMerge

private boolean skipForMerge(RegionStates regionStates,
                             RegionInfo regionInfo)

Determine if a RegionInfo should be considered for a merge operation.

computeMergeNormalizationPlans

private List<NormalizationPlan> computeMergeNormalizationPlans(SimpleRegionNormalizer.NormalizeContext ctx)

Computes the merge plans that should be executed for this table to converge average region towards target average or target region count.

skipForSplit

private static boolean skipForSplit(RegionState state,
                                    RegionInfo regionInfo)

Determine if a region in RegionState should be considered for a split operation.

computeSplitNormalizationPlans

private List<NormalizationPlan> computeSplitNormalizationPlans(SimpleRegionNormalizer.NormalizeContext ctx)

Computes the split plans that should be executed for this table to converge average region size towards target average or target region count.
if the region is > 2 times larger than average, we split it. split is more high priority normalization action than merge.

isOldEnoughForMerge

private boolean isOldEnoughForMerge(RegionInfo regionInfo)

Return true when regionInfo has a creation date that is old enough to be considered for a merge operation, false otherwise.

isLargeEnoughForMerge

private boolean isLargeEnoughForMerge(RegionInfo regionInfo)

Return true when regionInfo has a size that is sufficient to be considered for a merge operation, false otherwise.

logTraceReason

private static boolean logTraceReason(BooleanSupplier predicate,
                                      String fmtWhenTrue,
                                      Object... args)