org.apache.hadoop.hbase.quotas

Class FileArchiverNotifierImpl

java.lang.Object

org.apache.hadoop.hbase.quotas.FileArchiverNotifierImpl

All Implemented Interfaces:

FileArchiverNotifier

@InterfaceAudience.Private
public class FileArchiverNotifierImpl
extends Object
implements FileArchiverNotifier

Tracks file archiving and updates the hbase quota table.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	FileArchiverNotifierImpl.QuotaSnapshotSizeSerializationException An Exception thrown when SnapshotSize updates to hbase:quota fail to be written.
(package private) static class	FileArchiverNotifierImpl.SnapshotWithSize A struct encapsulating the name of a snapshot and its "size" on the filesystem.
(package private) static class	FileArchiverNotifierImpl.StoreFileReference A reference to a collection of files in the archive directory for a single region.

Field Summary

Fields

Modifier and Type	Field and Description
private org.apache.hadoop.conf.Configuration	conf
private Connection	conn
private List<String>	currentSnapshots
private org.apache.hadoop.fs.FileSystem	fs
private long	lastFullCompute
private static org.slf4j.Logger	LOG
private static Map<String,Object>	NAMESPACE_LOCKS
private ReentrantReadWriteLock.ReadLock	readLock
private TableName	tn
private ReentrantReadWriteLock.WriteLock	writeLock

Constructor Summary

Constructors

Constructor and Description
FileArchiverNotifierImpl(Connection conn, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, TableName tn)

Method Summary

Modifier and Type	Method and Description
void	addArchivedFiles(Set<Map.Entry<String,Long>> fileSizes) Records a file and its size in bytes being moved to the archive directory.
(package private) void	bucketFilesToSnapshot(String snapshotName, Map<String,Long> filesToUpdate, Map<String,Long> snapshotSizeChanges) For the given snapshot, find all files which this snapshotName references.
long	computeAndStoreSnapshotSizes(Collection<String> currentSnapshots) Computes the size of a table and all of its snapshots, recording new "full" sizes for each.
(package private) List<FileArchiverNotifierImpl.SnapshotWithSize>	computeSnapshotSizes(List<String> snapshots) Computes the size of each snapshot against the table referenced by this.
(package private) long	getLastFullCompute() Returns a strictly-increasing measure of time extracted by System.nanoTime().
(package private) static Object	getLockForNamespace(String namespace)
(package private) long	getPreviousNamespaceSnapshotSize(Table quotaTable, String namespace) Fetches the current size of all snapshots in the given namespace.
(package private) long	getSizeOfStoreFile(TableName tn, FileArchiverNotifierImpl.StoreFileReference storeFileName) Computes the size of the store files for a single region.
(package private) long	getSizeOfStoreFile(TableName tn, String regionName, String family, String storeFile) Computes the size of the store file given its name, region and family name in the archive directory.
(package private) long	getSizeOfStoreFiles(TableName tn, Set<FileArchiverNotifierImpl.StoreFileReference> storeFileNames) Computes the size of each store file in storeFileNames
(package private) long	getSnapshotSizeFromResult(Result r) Extracts the size component from a serialized SpaceQuotaSnapshot protobuf.
(package private) Set<FileArchiverNotifierImpl.StoreFileReference>	getStoreFilesFromSnapshot(SnapshotManifest manifest, Predicate<String> filter) Extracts the names of the store files referenced by this snapshot which satisfy the given predicate (the predicate returns true).
(package private) void	groupArchivedFiledBySnapshotAndRecordSize(List<String> snapshots, Set<Map.Entry<String,Long>> fileSizes) For each file in the map, this updates the first snapshot (lexicographic snapshot name) that references this file.
(package private) void	persistSnapshotSizeChanges(Map<String,Long> snapshotSizeChanges) Reads the current size for each snapshot to update, generates a new update based on that value, and then writes the new update.
(package private) void	persistSnapshotSizes(Table table, List<FileArchiverNotifierImpl.SnapshotWithSize> snapshotSizes) Writes the snapshot sizes to the provided table.
String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

LOG

private static final org.slf4j.Logger LOG

conn

private final Connection conn

conf

private final org.apache.hadoop.conf.Configuration conf

fs

private final org.apache.hadoop.fs.FileSystem fs

tn

private final TableName tn

readLock

private final ReentrantReadWriteLock.ReadLock readLock

writeLock

private final ReentrantReadWriteLock.WriteLock writeLock

lastFullCompute

private volatile long lastFullCompute

currentSnapshots

private List<String> currentSnapshots

NAMESPACE_LOCKS

private static final Map<String,Object> NAMESPACE_LOCKS

Constructor Detail

FileArchiverNotifierImpl

public FileArchiverNotifierImpl(Connection conn,
                                org.apache.hadoop.conf.Configuration conf,
                                org.apache.hadoop.fs.FileSystem fs,
                                TableName tn)

Method Detail

getLockForNamespace

static Object getLockForNamespace(String namespace)

getLastFullCompute

long getLastFullCompute()

Returns a strictly-increasing measure of time extracted by System.nanoTime().

addArchivedFiles

public void addArchivedFiles(Set<Map.Entry<String,Long>> fileSizes)
                      throws IOException

Description copied from interface: FileArchiverNotifier

Records a file and its size in bytes being moved to the archive directory.

Specified by:

addArchivedFiles in interface FileArchiverNotifier

Parameters:

fileSizes - A collection of file name to size in bytes

Throws:

IOException - If there was an IO-related error persisting the file size(s)

groupArchivedFiledBySnapshotAndRecordSize

void groupArchivedFiledBySnapshotAndRecordSize(List<String> snapshots,
                                               Set<Map.Entry<String,Long>> fileSizes)
                                        throws IOException

For each file in the map, this updates the first snapshot (lexicographic snapshot name) that references this file. The result of this computation is serialized to the quota table.

Parameters:

snapshots - A collection of HBase snapshots to group the files into

fileSizes - A map of file names to their sizes

Throws:

IOException

bucketFilesToSnapshot

void bucketFilesToSnapshot(String snapshotName,
                           Map<String,Long> filesToUpdate,
                           Map<String,Long> snapshotSizeChanges)
                    throws IOException

For the given snapshot, find all files which this snapshotName references. After a file is found to be referenced by the snapshot, it is removed from filesToUpdate and snapshotSizeChanges is updated in concert.

Parameters:

snapshotName - The snapshot to check

filesToUpdate - A mapping of archived files to their size

snapshotSizeChanges - A mapping of snapshots and their change in size

Throws:

IOException

persistSnapshotSizeChanges

void persistSnapshotSizeChanges(Map<String,Long> snapshotSizeChanges)
                         throws IOException

Reads the current size for each snapshot to update, generates a new update based on that value, and then writes the new update.

Parameters:

snapshotSizeChanges - A map of snapshot name to size change

Throws:

IOException

getPreviousNamespaceSnapshotSize

long getPreviousNamespaceSnapshotSize(Table quotaTable,
                                      String namespace)
                               throws IOException

Fetches the current size of all snapshots in the given namespace.

Parameters:

quotaTable - The HBase quota table

namespace - Namespace to fetch the sum of snapshot sizes for

Returns:

The size of all snapshot sizes for the namespace in bytes.

Throws:

IOException

getSnapshotSizeFromResult

long getSnapshotSizeFromResult(Result r)
                        throws org.apache.hbase.thirdparty.com.google.protobuf.InvalidProtocolBufferException

Extracts the size component from a serialized SpaceQuotaSnapshot protobuf.

Parameters:

r - A Result containing one cell with a SpaceQuotaSnapshot protobuf

Returns:

The size in bytes of the snapshot.

Throws:

org.apache.hbase.thirdparty.com.google.protobuf.InvalidProtocolBufferException

computeAndStoreSnapshotSizes

public long computeAndStoreSnapshotSizes(Collection<String> currentSnapshots)
                                  throws IOException

Description copied from interface: FileArchiverNotifier

Computes the size of a table and all of its snapshots, recording new "full" sizes for each.

Specified by:

computeAndStoreSnapshotSizes in interface FileArchiverNotifier

Parameters:

currentSnapshots - the current list of snapshots against this table

Returns:

The total size of all snapshots against this table.

Throws:

IOException - If there was an IO-related error computing or persisting the sizes.

toString

public String toString()

Overrides:

toString in class Object

computeSnapshotSizes

List<FileArchiverNotifierImpl.SnapshotWithSize> computeSnapshotSizes(List<String> snapshots)
                                                              throws IOException

Computes the size of each snapshot against the table referenced by this.

Parameters:

snapshots - A sorted list of snapshots against tn.

Returns:

A list of the size for each snapshot against tn.

Throws:

IOException

getSizeOfStoreFiles

long getSizeOfStoreFiles(TableName tn,
                         Set<FileArchiverNotifierImpl.StoreFileReference> storeFileNames)

Computes the size of each store file in storeFileNames

getSizeOfStoreFile

long getSizeOfStoreFile(TableName tn,
                        FileArchiverNotifierImpl.StoreFileReference storeFileName)

Computes the size of the store files for a single region.

getSizeOfStoreFile

long getSizeOfStoreFile(TableName tn,
                        String regionName,
                        String family,
                        String storeFile)

Computes the size of the store file given its name, region and family name in the archive directory.

getStoreFilesFromSnapshot

Set<FileArchiverNotifierImpl.StoreFileReference> getStoreFilesFromSnapshot(SnapshotManifest manifest,
                                                                           Predicate<String> filter)

Extracts the names of the store files referenced by this snapshot which satisfy the given predicate (the predicate returns true).

persistSnapshotSizes

void persistSnapshotSizes(Table table,
                          List<FileArchiverNotifierImpl.SnapshotWithSize> snapshotSizes)
                   throws IOException

Writes the snapshot sizes to the provided table.

Throws:

IOException