org.apache.hadoop.hbase.mob.compactions

Class PartitionedMobCompactor

java.lang.Object

org.apache.hadoop.hbase.mob.compactions.MobCompactor

org.apache.hadoop.hbase.mob.compactions.PartitionedMobCompactor

@InterfaceAudience.Private
public class PartitionedMobCompactor
extends MobCompactor

An implementation of MobCompactor that compacts the mob files in partitions.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
(package private) static class	PartitionedMobCompactor.DelPartitionComparator

Field Summary

Fields

Modifier and Type	Field and Description
private org.apache.hadoop.fs.Path	bulkloadPath
protected int	compactionBatchSize The number of files compacted in a batch
private CacheConfig	compactionCacheConfig
protected int	compactionKVMax
private Encryption.Context	cryptoContext
protected int	delFileMaxCount
private static org.slf4j.Logger	LOG
protected long	mergeableSize
private byte[]	refCellTags
private org.apache.hadoop.fs.Path	tempPath

Fields inherited from class org.apache.hadoop.hbase.mob.compactions.MobCompactor

column, conf, fs, mobFamilyDir, mobTableDir, pool, tableName

Constructor Summary

Constructors

Constructor and Description
PartitionedMobCompactor(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, TableName tableName, ColumnFamilyDescriptor column, ExecutorService pool)

Method Summary

Modifier and Type	Method and Description
private void	bulkloadRefFile(Connection connection, Table table, org.apache.hadoop.fs.Path bulkloadDirectory, String fileName) Bulkloads the current file.
private void	closeMobFileWriter(StoreFileWriter writer, long maxSeqId, long mobCellsCount) Closes the mob file writer.
private void	closeRefFileWriter(StoreFileWriter writer, long maxSeqId, long bulkloadTime) Closes the ref file writer.
private void	closeStoreFileReaders(List<HStoreFile> storeFiles) Closes the readers of store files.
List<org.apache.hadoop.fs.Path>	compact(List<org.apache.hadoop.fs.FileStatus> files, boolean allFiles) Compacts the candidate mob files.
protected List<org.apache.hadoop.fs.Path>	compactDelFiles(PartitionedMobCompactionRequest request, List<org.apache.hadoop.fs.Path> delFilePaths) Compacts the del files in batches which avoids opening too many files.
private org.apache.hadoop.fs.Path	compactDelFilesInBatch(PartitionedMobCompactionRequest request, List<HStoreFile> delFiles) Compacts the del file in a batch.
private List<org.apache.hadoop.fs.Path>	compactMobFilePartition(PartitionedMobCompactionRequest request, PartitionedMobCompactionRequest.CompactionPartition partition, List<HStoreFile> delFiles, Connection connection, Table table) Compacts a partition of selected small mob files and all the del files.
protected List<org.apache.hadoop.fs.Path>	compactMobFiles(PartitionedMobCompactionRequest request) Compacts the selected small mob files and all the del files.
private void	compactMobFilesInBatch(PartitionedMobCompactionRequest request, PartitionedMobCompactionRequest.CompactionPartition partition, Connection connection, Table table, List<HStoreFile> filesToCompact, int batch, org.apache.hadoop.fs.Path bulkloadPathOfPartition, org.apache.hadoop.fs.Path bulkloadColumnPath, List<org.apache.hadoop.fs.Path> newFiles) Compacts a partition of selected small mob files and all the del files in a batch.
private StoreScanner	createScanner(List<HStoreFile> filesToCompact, ScanType scanType) Creates a store scanner.
private void	deletePath(org.apache.hadoop.fs.Path path) Deletes a file.
private Pair<Long,Long>	getFileInfo(List<HStoreFile> storeFiles) Gets the max seqId and number of cells of the store files.
private org.apache.hadoop.fs.FileStatus	getLinkedFileStatus(HFileLink link)
(package private) List<HStoreFile>	getListOfDelFilesForPartition(PartitionedMobCompactionRequest.CompactionPartition partition, List<PartitionedMobCompactionRequest.CompactionDelPartition> delPartitions)
protected List<org.apache.hadoop.fs.Path>	performCompaction(PartitionedMobCompactionRequest request) Performs the compaction on the selected files.
protected PartitionedMobCompactionRequest	select(List<org.apache.hadoop.fs.FileStatus> candidates, boolean allFiles) Selects the compacted mob/del files.

Methods inherited from class org.apache.hadoop.hbase.mob.compactions.MobCompactor

compact, compact

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

private static final org.slf4j.Logger LOG

mergeableSize

protected long mergeableSize

delFileMaxCount

protected int delFileMaxCount

compactionBatchSize

protected int compactionBatchSize

The number of files compacted in a batch

compactionKVMax

protected int compactionKVMax

tempPath

private final org.apache.hadoop.fs.Path tempPath

bulkloadPath

private final org.apache.hadoop.fs.Path bulkloadPath

compactionCacheConfig

private final CacheConfig compactionCacheConfig

refCellTags

private final byte[] refCellTags

cryptoContext

private Encryption.Context cryptoContext

Constructor Detail

PartitionedMobCompactor

public PartitionedMobCompactor(org.apache.hadoop.conf.Configuration conf,
                               org.apache.hadoop.fs.FileSystem fs,
                               TableName tableName,
                               ColumnFamilyDescriptor column,
                               ExecutorService pool)
                        throws IOException

Throws:

IOException

Method Detail

compact

public List<org.apache.hadoop.fs.Path> compact(List<org.apache.hadoop.fs.FileStatus> files,
                                               boolean allFiles)
                                        throws IOException

Description copied from class: MobCompactor

Compacts the candidate mob files.

Specified by:

compact in class MobCompactor

Parameters:

files - The candidate mob files.

allFiles - Whether add all mob files into the compaction.

Returns:

The paths of new mob files generated in the compaction.

Throws:

IOException

select

protected PartitionedMobCompactionRequest select(List<org.apache.hadoop.fs.FileStatus> candidates,
                                                 boolean allFiles)
                                          throws IOException

Selects the compacted mob/del files. Iterates the candidates to find out all the del files and small mob files.

Parameters:

candidates - All the candidates.

allFiles - Whether add all mob files into the compaction.

Returns:

A compaction request.

Throws:

IOException - if IO failure is encountered

performCompaction

protected List<org.apache.hadoop.fs.Path> performCompaction(PartitionedMobCompactionRequest request)
                                                     throws IOException

Performs the compaction on the selected files.

1. Compacts the del files.
2. Compacts the selected small mob files and all the del files.
3. If all the candidates are selected, delete the del files.

Parameters:

request - The compaction request.

Returns:

The paths of new mob files generated in the compaction.

Throws:

IOException - if IO failure is encountered

getListOfDelFilesForPartition

List<HStoreFile> getListOfDelFilesForPartition(PartitionedMobCompactionRequest.CompactionPartition partition,
                                               List<PartitionedMobCompactionRequest.CompactionDelPartition> delPartitions)

compactMobFiles

protected List<org.apache.hadoop.fs.Path> compactMobFiles(PartitionedMobCompactionRequest request)
                                                   throws IOException

Compacts the selected small mob files and all the del files.

Parameters:

request - The compaction request.

Returns:

The paths of new mob files after compactions.

Throws:

IOException - if IO failure is encountered

compactMobFilePartition

private List<org.apache.hadoop.fs.Path> compactMobFilePartition(PartitionedMobCompactionRequest request,
                                                                PartitionedMobCompactionRequest.CompactionPartition partition,
                                                                List<HStoreFile> delFiles,
                                                                Connection connection,
                                                                Table table)
                                                         throws IOException

Compacts a partition of selected small mob files and all the del files.

Parameters:

request - The compaction request.

partition - A compaction partition.

delFiles - The del files.

connection - The connection to use.

table - The current table.

Returns:

The paths of new mob files after compactions.

Throws:

IOException - if IO failure is encountered

closeStoreFileReaders

private void closeStoreFileReaders(List<HStoreFile> storeFiles)

Closes the readers of store files.

Parameters:

storeFiles - The store files to be closed.

compactMobFilesInBatch

private void compactMobFilesInBatch(PartitionedMobCompactionRequest request,
                                    PartitionedMobCompactionRequest.CompactionPartition partition,
                                    Connection connection,
                                    Table table,
                                    List<HStoreFile> filesToCompact,
                                    int batch,
                                    org.apache.hadoop.fs.Path bulkloadPathOfPartition,
                                    org.apache.hadoop.fs.Path bulkloadColumnPath,
                                    List<org.apache.hadoop.fs.Path> newFiles)
                             throws IOException

Compacts a partition of selected small mob files and all the del files in a batch.

Parameters:

request - The compaction request.

partition - A compaction partition.

connection - To use for transport

table - The current table.

filesToCompact - The files to be compacted.

batch - The number of mob files to be compacted in a batch.

bulkloadPathOfPartition - The directory where the bulkload column of the current partition is saved.

bulkloadColumnPath - The directory where the bulkload files of current partition are saved.

newFiles - The paths of new mob files after compactions.

Throws:

IOException - if IO failure is encountered

compactDelFiles

protected List<org.apache.hadoop.fs.Path> compactDelFiles(PartitionedMobCompactionRequest request,
                                                          List<org.apache.hadoop.fs.Path> delFilePaths)
                                                   throws IOException

Compacts the del files in batches which avoids opening too many files.

Parameters:

request - The compaction request.

delFilePaths - Del file paths to compact

Returns:

The paths of new del files after merging or the original files if no merging is necessary.

Throws:

IOException - if IO failure is encountered

compactDelFilesInBatch

private org.apache.hadoop.fs.Path compactDelFilesInBatch(PartitionedMobCompactionRequest request,
                                                         List<HStoreFile> delFiles)
                                                  throws IOException

Compacts the del file in a batch.

Parameters:

request - The compaction request.

delFiles - The del files.

Returns:

The path of new del file after merging.

Throws:

IOException - if IO failure is encountered

createScanner

private StoreScanner createScanner(List<HStoreFile> filesToCompact,
                                   ScanType scanType)
                            throws IOException

Creates a store scanner.

Parameters:

filesToCompact - The files to be compacted.

scanType - The scan type.

Returns:

The store scanner.

Throws:

IOException - if IO failure is encountered

bulkloadRefFile

private void bulkloadRefFile(Connection connection,
                             Table table,
                             org.apache.hadoop.fs.Path bulkloadDirectory,
                             String fileName)
                      throws IOException

Bulkloads the current file.

Parameters:

connection - to use to get admin/RegionLocator

table - The current table.

bulkloadDirectory - The path of bulkload directory.

fileName - The current file name.

Throws:

IOException - if IO failure is encountered

closeMobFileWriter

private void closeMobFileWriter(StoreFileWriter writer,
                                long maxSeqId,
                                long mobCellsCount)
                         throws IOException

Closes the mob file writer.

Parameters:

writer - The mob file writer.

maxSeqId - Maximum sequence id.

mobCellsCount - The number of mob cells.

Throws:

IOException - if IO failure is encountered

closeRefFileWriter

private void closeRefFileWriter(StoreFileWriter writer,
                                long maxSeqId,
                                long bulkloadTime)
                         throws IOException

Closes the ref file writer.

Parameters:

writer - The ref file writer.

maxSeqId - Maximum sequence id.

bulkloadTime - The timestamp at which the bulk load file is created.

Throws:

IOException - if IO failure is encountered

getFileInfo

private Pair<Long,Long> getFileInfo(List<HStoreFile> storeFiles)
                             throws IOException

Gets the max seqId and number of cells of the store files.

Parameters:

storeFiles - The store files.

Returns:

The pair of the max seqId and number of cells of the store files.

Throws:

IOException - if IO failure is encountered

deletePath

private void deletePath(org.apache.hadoop.fs.Path path)

Deletes a file.

Parameters:

path - The path of the file to be deleted.

getLinkedFileStatus

private org.apache.hadoop.fs.FileStatus getLinkedFileStatus(HFileLink link)
                                                     throws IOException

Throws:

IOException