org.apache.hadoop.hbase.quotas

Class QuotaObserverChore

java.lang.Object

org.apache.hadoop.hbase.ScheduledChore

org.apache.hadoop.hbase.quotas.QuotaObserverChore

All Implemented Interfaces:

Runnable

@InterfaceAudience.Private
public class QuotaObserverChore
extends ScheduledChore

Reads the currently received Region filesystem-space use reports and acts on those which violate a defined quota.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
(package private) static class	QuotaObserverChore.TablesWithQuotas A container which encapsulates the tables that have either a table quota or are contained in a namespace which have a namespace quota.

Field Summary

Fields

Modifier and Type	Field and Description
private org.apache.hadoop.conf.Configuration	conf
private Connection	conn
private static org.slf4j.Logger	LOG
private MetricsMaster	metrics
private Map<String,SpaceQuotaSnapshot>	namespaceQuotaSnapshots
private QuotaSnapshotStore<String>	namespaceSnapshotStore
(package private) static long	QUOTA_OBSERVER_CHORE_DELAY_DEFAULT
(package private) static String	QUOTA_OBSERVER_CHORE_DELAY_KEY
(package private) static int	QUOTA_OBSERVER_CHORE_PERIOD_DEFAULT
(package private) static String	QUOTA_OBSERVER_CHORE_PERIOD_KEY
(package private) static double	QUOTA_OBSERVER_CHORE_REPORT_PERCENT_DEFAULT
(package private) static String	QUOTA_OBSERVER_CHORE_REPORT_PERCENT_KEY
(package private) static String	QUOTA_OBSERVER_CHORE_TIMEUNIT_DEFAULT
(package private) static String	QUOTA_OBSERVER_CHORE_TIMEUNIT_KEY
private MasterQuotaManager	quotaManager
private Map<String,SpaceQuotaSnapshot>	readOnlyNamespaceSnapshots
private Map<TableName,SpaceQuotaSnapshot>	readOnlyTableQuotaSnapshots
(package private) static long	REGION_REPORT_RETENTION_DURATION_DEFAULT
(package private) static String	REGION_REPORT_RETENTION_DURATION_KEY
private long	regionReportLifetimeMillis
private SpaceQuotaSnapshotNotifier	snapshotNotifier
private Map<TableName,SpaceQuotaSnapshot>	tableQuotaSnapshots
private QuotaSnapshotStore<TableName>	tableSnapshotStore

Constructor Summary

Constructors

Constructor and Description
QuotaObserverChore(Connection conn, org.apache.hadoop.conf.Configuration conf, SpaceQuotaSnapshotNotifier snapshotNotifier, MasterQuotaManager quotaManager, Stoppable stopper, MetricsMaster metrics)
QuotaObserverChore(HMaster master, MetricsMaster metrics)

Method Summary

Modifier and Type	Method and Description
(package private) void	_chore()
protected void	chore() The task to execute on each scheduled execution of the Chore
(package private) QuotaObserverChore.TablesWithQuotas	fetchAllTablesWithQuotasDefined() Computes the set of all tables that have quotas defined.
(package private) static long	getInitialDelay(org.apache.hadoop.conf.Configuration conf) Extracts the initial delay for the chore from the configuration.
(package private) SpaceQuotaSnapshot	getNamespaceQuotaSnapshot(String namespace) Fetches the SpaceQuotaSnapshot for the given namespace from this chore.
Map<String,SpaceQuotaSnapshot>	getNamespaceQuotaSnapshots() Returns an unmodifiable view over the current SpaceQuotaSnapshot objects for each HBase namespace with a quota defined.
(package private) QuotaSnapshotStore<String>	getNamespaceSnapshotStore()
(package private) static int	getPeriod(org.apache.hadoop.conf.Configuration conf) Extracts the period for the chore from the configuration.
(package private) static Double	getRegionReportPercent(org.apache.hadoop.conf.Configuration conf) Extracts the percent of Regions for a table to have been reported to enable quota violation state change.
(package private) SpaceQuotaSnapshot	getTableQuotaSnapshot(TableName table) Fetches the SpaceQuotaSnapshot for the given table.
Map<TableName,SpaceQuotaSnapshot>	getTableQuotaSnapshots() Returns an unmodifiable view over the current SpaceQuotaSnapshot objects for each HBase table with a quota defined.
(package private) QuotaSnapshotStore<TableName>	getTableSnapshotStore()
(package private) static TimeUnit	getTimeUnit(org.apache.hadoop.conf.Configuration conf) Extracts the time unit for the chore period and initial delay from the configuration.
(package private) void	initializeSnapshotStores(Map<RegionInfo,Long> regionSizes)
private boolean	isDisableSpaceViolationPolicy(SpaceViolationPolicy currPolicy, SpaceViolationPolicy targetPolicy) Method to check whether we are dealing with DISABLE SpaceViolationPolicy.
(package private) void	processNamespacesWithQuotas(Set<String> namespacesWithQuotas, org.apache.hbase.thirdparty.com.google.common.collect.Multimap<String,TableName> tablesByNamespace) Processes each namespace which has a quota defined and moves all of the tables contained in that namespace into or out of violation of the quota.
(package private) void	processTablesWithQuotas(Set<TableName> tablesWithTableQuotas) Processes each TableName which has a quota defined and moves it in or out of violation based on the space use.
(package private) void	pruneOldRegionReports() Removes region reports over a certain age.
(package private) void	setNamespaceQuotaSnapshot(String namespace, SpaceQuotaSnapshot snapshot) Stores the given snapshot for the given namespace in this chore.
(package private) void	setTableQuotaSnapshot(TableName table, SpaceQuotaSnapshot snapshot) Stores the quota state for the given table.
(package private) void	updateNamespaceQuota(String namespace, SpaceQuotaSnapshot currentSnapshot, SpaceQuotaSnapshot targetSnapshot, org.apache.hbase.thirdparty.com.google.common.collect.Multimap<String,TableName> tablesByNamespace) Updates the hbase:quota table with the target quota policy for this namespace if necessary.
(package private) void	updateTableQuota(TableName table, SpaceQuotaSnapshot currentSnapshot, SpaceQuotaSnapshot targetSnapshot) Updates the hbase:quota table with the new quota policy for this table if necessary.

Methods inherited from class org.apache.hadoop.hbase.ScheduledChore

cancel, cancel, choreForTesting, cleanup, getInitialDelay, getName, getPeriod, getStopper, getTimeUnit, initialChore, isInitialChoreComplete, isScheduled, run, shutdown, shutdown, toString, triggerNow

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

LOG

private static final org.slf4j.Logger LOG

QUOTA_OBSERVER_CHORE_PERIOD_KEY

static final String QUOTA_OBSERVER_CHORE_PERIOD_KEY

See Also:

Constant Field Values

QUOTA_OBSERVER_CHORE_PERIOD_DEFAULT

static final int QUOTA_OBSERVER_CHORE_PERIOD_DEFAULT

See Also:

Constant Field Values

QUOTA_OBSERVER_CHORE_DELAY_KEY

static final String QUOTA_OBSERVER_CHORE_DELAY_KEY

See Also:

Constant Field Values

QUOTA_OBSERVER_CHORE_DELAY_DEFAULT

static final long QUOTA_OBSERVER_CHORE_DELAY_DEFAULT

See Also:

Constant Field Values

QUOTA_OBSERVER_CHORE_TIMEUNIT_KEY

static final String QUOTA_OBSERVER_CHORE_TIMEUNIT_KEY

See Also:

Constant Field Values

QUOTA_OBSERVER_CHORE_TIMEUNIT_DEFAULT

static final String QUOTA_OBSERVER_CHORE_TIMEUNIT_DEFAULT

QUOTA_OBSERVER_CHORE_REPORT_PERCENT_KEY

static final String QUOTA_OBSERVER_CHORE_REPORT_PERCENT_KEY

See Also:

Constant Field Values

QUOTA_OBSERVER_CHORE_REPORT_PERCENT_DEFAULT

static final double QUOTA_OBSERVER_CHORE_REPORT_PERCENT_DEFAULT

See Also:

Constant Field Values

REGION_REPORT_RETENTION_DURATION_KEY

static final String REGION_REPORT_RETENTION_DURATION_KEY

See Also:

Constant Field Values

REGION_REPORT_RETENTION_DURATION_DEFAULT

static final long REGION_REPORT_RETENTION_DURATION_DEFAULT

See Also:

Constant Field Values

conn

private final Connection conn

conf

private final org.apache.hadoop.conf.Configuration conf

quotaManager

private final MasterQuotaManager quotaManager

metrics

private final MetricsMaster metrics

snapshotNotifier

private final SpaceQuotaSnapshotNotifier snapshotNotifier

tableQuotaSnapshots

private final Map<TableName,SpaceQuotaSnapshot> tableQuotaSnapshots

readOnlyTableQuotaSnapshots

private final Map<TableName,SpaceQuotaSnapshot> readOnlyTableQuotaSnapshots

namespaceQuotaSnapshots

private final Map<String,SpaceQuotaSnapshot> namespaceQuotaSnapshots

readOnlyNamespaceSnapshots

private final Map<String,SpaceQuotaSnapshot> readOnlyNamespaceSnapshots

regionReportLifetimeMillis

private final long regionReportLifetimeMillis

tableSnapshotStore

private QuotaSnapshotStore<TableName> tableSnapshotStore

namespaceSnapshotStore

private QuotaSnapshotStore<String> namespaceSnapshotStore

Constructor Detail

QuotaObserverChore

public QuotaObserverChore(HMaster master,
                          MetricsMaster metrics)

QuotaObserverChore

QuotaObserverChore(Connection conn,
                   org.apache.hadoop.conf.Configuration conf,
                   SpaceQuotaSnapshotNotifier snapshotNotifier,
                   MasterQuotaManager quotaManager,
                   Stoppable stopper,
                   MetricsMaster metrics)

Method Detail

chore

protected void chore()

Description copied from class: ScheduledChore

The task to execute on each scheduled execution of the Chore

Specified by:

chore in class ScheduledChore

_chore

void _chore()
     throws IOException

Throws:

IOException

initializeSnapshotStores

void initializeSnapshotStores(Map<RegionInfo,Long> regionSizes)

processTablesWithQuotas

void processTablesWithQuotas(Set<TableName> tablesWithTableQuotas)
                      throws IOException

Processes each TableName which has a quota defined and moves it in or out of violation based on the space use.

Parameters:

tablesWithTableQuotas - The HBase tables which have quotas defined

Throws:

IOException

processNamespacesWithQuotas

void processNamespacesWithQuotas(Set<String> namespacesWithQuotas,
                                 org.apache.hbase.thirdparty.com.google.common.collect.Multimap<String,TableName> tablesByNamespace)
                          throws IOException

Processes each namespace which has a quota defined and moves all of the tables contained in that namespace into or out of violation of the quota. Tables which are already in violation of a quota at the table level which also have a reside in a namespace with a violated quota will not have the namespace quota enacted. The table quota takes priority over the namespace quota.

Parameters:

namespacesWithQuotas - The set of namespaces that have quotas defined

tablesByNamespace - A mapping of namespaces and the tables contained in those namespaces

Throws:

IOException

updateTableQuota

void updateTableQuota(TableName table,
                      SpaceQuotaSnapshot currentSnapshot,
                      SpaceQuotaSnapshot targetSnapshot)
               throws IOException

Updates the hbase:quota table with the new quota policy for this table if necessary.

Parameters:

table - The table being checked

currentSnapshot - The state of the quota on this table from the previous invocation.

targetSnapshot - The state the quota should be in for this table.

Throws:

IOException

isDisableSpaceViolationPolicy

private boolean isDisableSpaceViolationPolicy(SpaceViolationPolicy currPolicy,
                                              SpaceViolationPolicy targetPolicy)

Method to check whether we are dealing with DISABLE SpaceViolationPolicy. In such a case, currPolicy or/and targetPolicy will be having DISABLE policy.

Parameters:

currPolicy - currently set space violation policy

targetPolicy - new space violation policy

Returns:

true if is DISABLE space violation policy; otherwise false

updateNamespaceQuota

void updateNamespaceQuota(String namespace,
                          SpaceQuotaSnapshot currentSnapshot,
                          SpaceQuotaSnapshot targetSnapshot,
                          org.apache.hbase.thirdparty.com.google.common.collect.Multimap<String,TableName> tablesByNamespace)
                   throws IOException

Updates the hbase:quota table with the target quota policy for this namespace if necessary.

Parameters:

namespace - The namespace being checked

currentSnapshot - The state of the quota on this namespace from the previous invocation

targetSnapshot - The state the quota should be in for this namespace

tablesByNamespace - A mapping of tables in namespaces.

Throws:

IOException

pruneOldRegionReports

void pruneOldRegionReports()

Removes region reports over a certain age.

fetchAllTablesWithQuotasDefined

QuotaObserverChore.TablesWithQuotas fetchAllTablesWithQuotasDefined()
                                                             throws IOException

Computes the set of all tables that have quotas defined. This includes tables with quotas explicitly set on them, in addition to tables that exist namespaces which have a quota defined.

Throws:

IOException

getTableSnapshotStore

QuotaSnapshotStore<TableName> getTableSnapshotStore()

getNamespaceSnapshotStore

QuotaSnapshotStore<String> getNamespaceSnapshotStore()

getTableQuotaSnapshots

public Map<TableName,SpaceQuotaSnapshot> getTableQuotaSnapshots()

Returns an unmodifiable view over the current SpaceQuotaSnapshot objects for each HBase table with a quota defined.

getNamespaceQuotaSnapshots

public Map<String,SpaceQuotaSnapshot> getNamespaceQuotaSnapshots()

Returns an unmodifiable view over the current SpaceQuotaSnapshot objects for each HBase namespace with a quota defined.

getTableQuotaSnapshot

SpaceQuotaSnapshot getTableQuotaSnapshot(TableName table)

Fetches the SpaceQuotaSnapshot for the given table.

setTableQuotaSnapshot

void setTableQuotaSnapshot(TableName table,
                           SpaceQuotaSnapshot snapshot)

Stores the quota state for the given table.

getNamespaceQuotaSnapshot

SpaceQuotaSnapshot getNamespaceQuotaSnapshot(String namespace)

Fetches the SpaceQuotaSnapshot for the given namespace from this chore.

setNamespaceQuotaSnapshot

void setNamespaceQuotaSnapshot(String namespace,
                               SpaceQuotaSnapshot snapshot)

Stores the given snapshot for the given namespace in this chore.

getPeriod

static int getPeriod(org.apache.hadoop.conf.Configuration conf)

Extracts the period for the chore from the configuration.

Parameters:

conf - The configuration object.

Returns:

The configured chore period or the default value in the given timeunit.

See Also:

getTimeUnit(Configuration)

getInitialDelay

static long getInitialDelay(org.apache.hadoop.conf.Configuration conf)

Extracts the initial delay for the chore from the configuration.

Parameters:

conf - The configuration object.

Returns:

The configured chore initial delay or the default value in the given timeunit.

See Also:

getTimeUnit(Configuration)

getTimeUnit

static TimeUnit getTimeUnit(org.apache.hadoop.conf.Configuration conf)

Extracts the time unit for the chore period and initial delay from the configuration. The configuration value for QUOTA_OBSERVER_CHORE_TIMEUNIT_KEY must correspond to a TimeUnit value.

Parameters:

conf - The configuration object.

Returns:

The configured time unit for the chore period and initial delay or the default value.

getRegionReportPercent

static Double getRegionReportPercent(org.apache.hadoop.conf.Configuration conf)

Extracts the percent of Regions for a table to have been reported to enable quota violation state change.

Parameters:

conf - The configuration object.

Returns:

The percent of regions reported to use.