org.apache.hadoop.hbase.quotas

Class SnapshotQuotaObserverChore

java.lang.Object

org.apache.hadoop.hbase.ScheduledChore

org.apache.hadoop.hbase.quotas.SnapshotQuotaObserverChore

All Implemented Interfaces:

Runnable

@InterfaceAudience.Private
public class SnapshotQuotaObserverChore
extends ScheduledChore

A Master-invoked Chore that computes the size of each snapshot which was created from a table which has a space quota.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
(package private) static class	SnapshotQuotaObserverChore.SnapshotWithSize A struct encapsulating the name of a snapshot and its "size" on the filesystem.
(package private) static class	SnapshotQuotaObserverChore.StoreFileReference A reference to a collection of files in the archive directory for a single region.

Field Summary

Fields

Modifier and Type	Field and Description
private org.apache.hadoop.conf.Configuration	conf
private Connection	conn
private org.apache.hadoop.fs.FileSystem	fs
private static org.slf4j.Logger	LOG
private MetricsMaster	metrics
(package private) static long	SNAPSHOT_QUOTA_CHORE_DELAY_DEFAULT
(package private) static String	SNAPSHOT_QUOTA_CHORE_DELAY_KEY
(package private) static int	SNAPSHOT_QUOTA_CHORE_PERIOD_DEFAULT
(package private) static String	SNAPSHOT_QUOTA_CHORE_PERIOD_KEY
(package private) static String	SNAPSHOT_QUOTA_CHORE_TIMEUNIT_DEFAULT
(package private) static String	SNAPSHOT_QUOTA_CHORE_TIMEUNIT_KEY

Constructor Summary

Constructors

Constructor and Description
SnapshotQuotaObserverChore(Connection conn, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, Stoppable stopper, MetricsMaster metrics)
SnapshotQuotaObserverChore(HMaster master, MetricsMaster metrics)

Method Summary

Modifier and Type	Method and Description
(package private) void	_chore()
protected void	chore() The task to execute on each scheduled execution of the Chore
(package private) org.apache.hbase.thirdparty.com.google.common.collect.Multimap<TableName,SnapshotQuotaObserverChore.SnapshotWithSize>	computeSnapshotSizes(org.apache.hbase.thirdparty.com.google.common.collect.Multimap<TableName,String> snapshotsToComputeSize) Computes the size of each snapshot provided given the current files referenced by the table.
(package private) static long	getInitialDelay(org.apache.hadoop.conf.Configuration conf) Extracts the initial delay for the chore from the configuration.
(package private) static int	getPeriod(org.apache.hadoop.conf.Configuration conf) Extracts the period for the chore from the configuration.
(package private) long	getSizeOfStoreFile(TableName tn, SnapshotQuotaObserverChore.StoreFileReference storeFileName) Computes the size of the store files for a single region.
(package private) long	getSizeOfStoreFile(TableName tn, String regionName, String family, String storeFile) Computes the size of the store file given its name, region and family name in the archive directory.
(package private) long	getSizeOfStoreFiles(TableName tn, Set<SnapshotQuotaObserverChore.StoreFileReference> storeFileNames) Computes the size of each store file in storeFileNames
(package private) org.apache.hbase.thirdparty.com.google.common.collect.Multimap<TableName,String>	getSnapshotsFromTables(Admin admin, Set<TableName> tablesToFetchSnapshotsFrom) Computes a mapping of originating TableName to snapshots, when the TableName exists in the provided Set.
(package private) org.apache.hbase.thirdparty.com.google.common.collect.Multimap<TableName,String>	getSnapshotsToComputeSize() Fetches each table with a quota (table or namespace quota), and then fetch the name of each snapshot which was created from that table.
(package private) Set<SnapshotQuotaObserverChore.StoreFileReference>	getStoreFilesFromSnapshot(SnapshotManifest manifest, Predicate<String> filter) Extracts the names of the store files referenced by this snapshot which satisfy the given predicate (the predicate returns true).
(package private) org.apache.hadoop.fs.Path	getTableDir(TableName tn) Calculates the directory in HDFS for a table based on the configuration.
(package private) static TimeUnit	getTimeUnit(org.apache.hadoop.conf.Configuration conf) Extracts the time unit for the chore period and initial delay from the configuration.
(package private) Map<String,Long>	groupSnapshotSizesByNamespace(org.apache.hbase.thirdparty.com.google.common.collect.Multimap<TableName,SnapshotQuotaObserverChore.SnapshotWithSize> snapshotsWithSize) Sums the snapshot sizes for each namespace.
(package private) void	persistSnapshotSizes(org.apache.hbase.thirdparty.com.google.common.collect.Multimap<TableName,SnapshotQuotaObserverChore.SnapshotWithSize> snapshotsWithSize) Writes the snapshot sizes to the hbase:quota table.
(package private) void	persistSnapshotSizes(Table table, org.apache.hbase.thirdparty.com.google.common.collect.Multimap<TableName,SnapshotQuotaObserverChore.SnapshotWithSize> snapshotsWithSize) Writes the snapshot sizes to the provided table.
(package private) void	persistSnapshotSizesByNS(Table quotaTable, org.apache.hbase.thirdparty.com.google.common.collect.Multimap<TableName,SnapshotQuotaObserverChore.SnapshotWithSize> snapshotsWithSize) Rolls up the snapshot sizes by namespace and writes a single record for each namespace which is the size of all snapshots in that namespace.

Methods inherited from class org.apache.hadoop.hbase.ScheduledChore

cancel, cancel, choreForTesting, cleanup, getInitialDelay, getName, getPeriod, getStopper, getTimeUnit, initialChore, isInitialChoreComplete, isScheduled, run, toString, triggerNow

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

LOG

private static final org.slf4j.Logger LOG

SNAPSHOT_QUOTA_CHORE_PERIOD_KEY

static final String SNAPSHOT_QUOTA_CHORE_PERIOD_KEY

See Also:

Constant Field Values

SNAPSHOT_QUOTA_CHORE_PERIOD_DEFAULT

static final int SNAPSHOT_QUOTA_CHORE_PERIOD_DEFAULT

See Also:

Constant Field Values

SNAPSHOT_QUOTA_CHORE_DELAY_KEY

static final String SNAPSHOT_QUOTA_CHORE_DELAY_KEY

See Also:

Constant Field Values

SNAPSHOT_QUOTA_CHORE_DELAY_DEFAULT

static final long SNAPSHOT_QUOTA_CHORE_DELAY_DEFAULT

See Also:

Constant Field Values

SNAPSHOT_QUOTA_CHORE_TIMEUNIT_KEY

static final String SNAPSHOT_QUOTA_CHORE_TIMEUNIT_KEY

See Also:

Constant Field Values

SNAPSHOT_QUOTA_CHORE_TIMEUNIT_DEFAULT

static final String SNAPSHOT_QUOTA_CHORE_TIMEUNIT_DEFAULT

conn

private final Connection conn

conf

private final org.apache.hadoop.conf.Configuration conf

metrics

private final MetricsMaster metrics

fs

private final org.apache.hadoop.fs.FileSystem fs

Constructor Detail

SnapshotQuotaObserverChore

public SnapshotQuotaObserverChore(HMaster master,
                                  MetricsMaster metrics)

SnapshotQuotaObserverChore

SnapshotQuotaObserverChore(Connection conn,
                           org.apache.hadoop.conf.Configuration conf,
                           org.apache.hadoop.fs.FileSystem fs,
                           Stoppable stopper,
                           MetricsMaster metrics)

Method Detail

chore

protected void chore()

Description copied from class: ScheduledChore

The task to execute on each scheduled execution of the Chore

Specified by:

chore in class ScheduledChore

_chore

void _chore()
     throws IOException

Throws:

IOException

getSnapshotsToComputeSize

org.apache.hbase.thirdparty.com.google.common.collect.Multimap<TableName,String> getSnapshotsToComputeSize()
                                                                                                    throws IOException

Fetches each table with a quota (table or namespace quota), and then fetch the name of each snapshot which was created from that table.

Returns:

A mapping of table to snapshots created from that table

Throws:

IOException

getSnapshotsFromTables

org.apache.hbase.thirdparty.com.google.common.collect.Multimap<TableName,String> getSnapshotsFromTables(Admin admin,
                                                                                                        Set<TableName> tablesToFetchSnapshotsFrom)
                                                                                                 throws IOException

Computes a mapping of originating TableName to snapshots, when the TableName exists in the provided Set.

Throws:

IOException

computeSnapshotSizes

org.apache.hbase.thirdparty.com.google.common.collect.Multimap<TableName,SnapshotQuotaObserverChore.SnapshotWithSize> computeSnapshotSizes(org.apache.hbase.thirdparty.com.google.common.collect.Multimap<TableName,String> snapshotsToComputeSize)
                                                                                                                                    throws IOException

Computes the size of each snapshot provided given the current files referenced by the table.

Parameters:

snapshotsToComputeSize - The snapshots to compute the size of

Returns:

A mapping of table to snapshot created from that table and the snapshot's size.

Throws:

IOException

getStoreFilesFromSnapshot

Set<SnapshotQuotaObserverChore.StoreFileReference> getStoreFilesFromSnapshot(SnapshotManifest manifest,
                                                                             Predicate<String> filter)

Extracts the names of the store files referenced by this snapshot which satisfy the given predicate (the predicate returns true).

getTableDir

org.apache.hadoop.fs.Path getTableDir(TableName tn)
                               throws IOException

Calculates the directory in HDFS for a table based on the configuration.

Throws:

IOException

getSizeOfStoreFiles

long getSizeOfStoreFiles(TableName tn,
                         Set<SnapshotQuotaObserverChore.StoreFileReference> storeFileNames)

Computes the size of each store file in storeFileNames

getSizeOfStoreFile

long getSizeOfStoreFile(TableName tn,
                        SnapshotQuotaObserverChore.StoreFileReference storeFileName)

Computes the size of the store files for a single region.

getSizeOfStoreFile

long getSizeOfStoreFile(TableName tn,
                        String regionName,
                        String family,
                        String storeFile)

Computes the size of the store file given its name, region and family name in the archive directory.

persistSnapshotSizes

void persistSnapshotSizes(org.apache.hbase.thirdparty.com.google.common.collect.Multimap<TableName,SnapshotQuotaObserverChore.SnapshotWithSize> snapshotsWithSize)
                   throws IOException

Writes the snapshot sizes to the hbase:quota table.

Parameters:

snapshotsWithSize - The snapshot sizes to write.

Throws:

IOException

persistSnapshotSizes

void persistSnapshotSizes(Table table,
                          org.apache.hbase.thirdparty.com.google.common.collect.Multimap<TableName,SnapshotQuotaObserverChore.SnapshotWithSize> snapshotsWithSize)
                   throws IOException

Writes the snapshot sizes to the provided table.

Throws:

IOException

persistSnapshotSizesByNS

void persistSnapshotSizesByNS(Table quotaTable,
                              org.apache.hbase.thirdparty.com.google.common.collect.Multimap<TableName,SnapshotQuotaObserverChore.SnapshotWithSize> snapshotsWithSize)
                       throws IOException

Rolls up the snapshot sizes by namespace and writes a single record for each namespace which is the size of all snapshots in that namespace.

Throws:

IOException

groupSnapshotSizesByNamespace

Map<String,Long> groupSnapshotSizesByNamespace(org.apache.hbase.thirdparty.com.google.common.collect.Multimap<TableName,SnapshotQuotaObserverChore.SnapshotWithSize> snapshotsWithSize)

Sums the snapshot sizes for each namespace.

getPeriod

static int getPeriod(org.apache.hadoop.conf.Configuration conf)

Extracts the period for the chore from the configuration.

Parameters:

conf - The configuration object.

Returns:

The configured chore period or the default value.

getInitialDelay

static long getInitialDelay(org.apache.hadoop.conf.Configuration conf)

Extracts the initial delay for the chore from the configuration.

Parameters:

conf - The configuration object.

Returns:

The configured chore initial delay or the default value.

getTimeUnit

static TimeUnit getTimeUnit(org.apache.hadoop.conf.Configuration conf)

Extracts the time unit for the chore period and initial delay from the configuration. The configuration value for SNAPSHOT_QUOTA_CHORE_TIMEUNIT_KEY must correspond to a TimeUnit value.

Parameters:

conf - The configuration object.

Returns:

The configured time unit for the chore period and initial delay or the default value.