org.apache.hadoop.hbase.mob

Class MobUtils

java.lang.Object

org.apache.hadoop.hbase.mob.MobUtils

@InterfaceAudience.Private
public final class MobUtils
extends Object

The mob utilities

Field Summary

Fields

Modifier and Type	Field and Description
private static ThreadLocal<SimpleDateFormat>	LOCAL_FORMAT
private static org.slf4j.Logger	LOG
private static long	MONTHLY_THRESHOLD_MULTIPLIER
private static byte[]	REF_DELETE_MARKER_TAG_BYTES
private static long	WEEKLY_THRESHOLD_MULTIPLIER

Constructor Summary

Constructors

Modifier	Constructor and Description
private	MobUtils() Private constructor to keep this class from being instantiated.

Method Summary

Modifier and Type	Method and Description
static void	cleanExpiredMobFiles(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.conf.Configuration conf, TableName tableName, ColumnFamilyDescriptor columnDescriptor, CacheConfig cacheConfig, long current) Cleans the expired mob files.
static org.apache.hadoop.fs.Path	commitFile(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path sourceFile, org.apache.hadoop.fs.Path targetPath, CacheConfig cacheConfig) Commits the mob file.
static StoreFileWriter	createDelFileWriter(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, ColumnFamilyDescriptor family, String date, org.apache.hadoop.fs.Path basePath, long maxKeyCount, Compression.Algorithm compression, byte[] startKey, CacheConfig cacheConfig, Encryption.Context cryptoContext) Creates a writer for the del file in temp directory.
static ExecutorService	createMobCompactorThreadPool(org.apache.hadoop.conf.Configuration conf) Creates a thread pool.
static Cell	createMobRefCell(Cell cell, byte[] fileName, byte[] refCellTags)
static Cell	createMobRefCell(Cell cell, byte[] fileName, Tag tableNameTag) Creates a mob reference KeyValue.
static Cell	createMobRefDeleteMarker(Cell cell) Creates a mob ref delete marker.
static StoreFileWriter	createRefFileWriter(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, ColumnFamilyDescriptor family, org.apache.hadoop.fs.Path basePath, long maxKeyCount, CacheConfig cacheConfig, Encryption.Context cryptoContext, boolean isCompaction) Creates a writer for the ref file in temp directory.
static StoreFileWriter	createWriter(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, ColumnFamilyDescriptor family, MobFileName mobFileName, org.apache.hadoop.fs.Path basePath, long maxKeyCount, Compression.Algorithm compression, CacheConfig cacheConfig, Encryption.Context cryptoContext, boolean isCompaction) Creates a writer for the mob file in temp directory.
static StoreFileWriter	createWriter(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, ColumnFamilyDescriptor family, org.apache.hadoop.fs.Path path, long maxKeyCount, Compression.Algorithm compression, CacheConfig cacheConfig, Encryption.Context cryptoContext, ChecksumType checksumType, int bytesPerChecksum, int blocksize, BloomType bloomType, boolean isCompaction) Creates a writer for the mob file in temp directory.
static StoreFileWriter	createWriter(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, ColumnFamilyDescriptor family, String date, org.apache.hadoop.fs.Path basePath, long maxKeyCount, Compression.Algorithm compression, byte[] startKey, CacheConfig cacheConfig, Encryption.Context cryptoContext, boolean isCompaction) Creates a writer for the mob file in temp directory.
static StoreFileWriter	createWriter(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, ColumnFamilyDescriptor family, String date, org.apache.hadoop.fs.Path basePath, long maxKeyCount, Compression.Algorithm compression, String startKey, CacheConfig cacheConfig, Encryption.Context cryptoContext, boolean isCompaction) Creates a writer for the mob file in temp directory.
static void	doMobCompaction(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, TableName tableName, ColumnFamilyDescriptor hcd, ExecutorService pool, boolean allFiles, LockManager.MasterLock lock) Performs the mob compaction.
static boolean	fillPartitionId(PartitionedMobCompactionRequest.CompactionPartitionId id, Date firstDayOfCurrentMonth, Date firstDayOfCurrentWeek, String dateStr, MobCompactPartitionPolicy policy, Calendar calendar, long threshold) fill out partition id based on compaction policy and date, threshold...
static String	formatDate(Date date) Formats a date to a string.
static org.apache.hadoop.fs.Path	getCompactionWorkingPath(org.apache.hadoop.fs.Path root, String jobName) Gets the working directory of the mob compaction.
static Date	getFirstDayOfMonth(Calendar calendar, Date date) Get the first day of the input date's month
static Date	getFirstDayOfWeek(Calendar calendar, Date date) Get the first day of the input date's week
static org.apache.hadoop.fs.Path	getMobFamilyPath(org.apache.hadoop.conf.Configuration conf, TableName tableName, String familyName) Gets the family dir of the mob files.
static org.apache.hadoop.fs.Path	getMobFamilyPath(org.apache.hadoop.fs.Path regionPath, String familyName) Gets the family dir of the mob files.
static String	getMobFileName(Cell cell) Gets the mob file name from the mob ref cell.
static org.apache.hadoop.fs.Path	getMobHome(org.apache.hadoop.conf.Configuration conf) Gets the root dir of the mob files.
static org.apache.hadoop.fs.Path	getMobHome(org.apache.hadoop.fs.Path rootDir) Gets the root dir of the mob files under the qualified HBase root dir.
static RegionInfo	getMobRegionInfo(TableName tableName) Gets the RegionInfo of the mob files.
static org.apache.hadoop.fs.Path	getMobRegionPath(org.apache.hadoop.conf.Configuration conf, TableName tableName) Gets the region dir of the mob files.
static org.apache.hadoop.fs.Path	getMobRegionPath(org.apache.hadoop.fs.Path rootDir, TableName tableName) Gets the region dir of the mob files under the specified root dir.
static org.apache.hadoop.fs.Path	getMobTableDir(org.apache.hadoop.fs.Path rootDir, TableName tableName) Gets the table dir of the mob files under the qualified HBase root dir.
static int	getMobValueLength(Cell cell) Gets the mob value length from the mob ref cell.
static org.apache.hadoop.fs.Path	getQualifiedMobRootDir(org.apache.hadoop.conf.Configuration conf) Gets the qualified root dir of the mob files.
static TableName	getTableLockName(TableName tn) Gets the table name used in the table lock.
static Tag	getTableNameTag(Cell cell) Gets the table name tag.
static boolean	hasMobColumns(TableDescriptor htd) Checks whether this table has mob-enabled columns.
static boolean	hasMobReferenceTag(List<Tag> tags) Whether the tag list has a mob reference tag.
static boolean	hasValidMobRefCellValue(Cell cell) Indicates whether the current mob ref cell has a valid value.
static boolean	isCacheMobBlocks(Scan scan) Indicates whether the scan contains the information of caching blocks.
static boolean	isMobFileExpired(ColumnFamilyDescriptor column, long current, String fileDate) Checks if the mob file is expired.
static boolean	isMobReferenceCell(Cell cell) Whether the current cell is a mob reference cell.
static boolean	isMobRegionInfo(RegionInfo regionInfo) Gets whether the current RegionInfo is a mob one.
static boolean	isMobRegionName(TableName tableName, byte[] regionName) Gets whether the current region name follows the pattern of a mob region name.
static boolean	isRawMobScan(Scan scan) Indicates whether it's a raw scan.
static boolean	isReadEmptyValueOnMobCellMiss(Scan scan) Indicates whether return null value when the mob file is missing or corrupt.
static boolean	isRefOnlyScan(Scan scan) Indicates whether it's a reference only scan.
static Date	parseDate(String dateString) Parses the string to a date.
static void	removeMobFiles(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, TableName tableName, org.apache.hadoop.fs.Path tableDir, byte[] family, Collection<HStoreFile> storeFiles) Archives the mob files.
static void	setCacheMobBlocks(Scan scan, boolean cacheBlocks) Sets the attribute of caching blocks in the scan.
private static void	validateMobFile(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, CacheConfig cacheConfig, boolean primaryReplica) Validates a mob file by opening and closing it.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

private static final org.slf4j.Logger LOG

WEEKLY_THRESHOLD_MULTIPLIER

private static final long WEEKLY_THRESHOLD_MULTIPLIER

See Also:

Constant Field Values

MONTHLY_THRESHOLD_MULTIPLIER

private static final long MONTHLY_THRESHOLD_MULTIPLIER

See Also:

Constant Field Values

LOCAL_FORMAT

private static final ThreadLocal<SimpleDateFormat> LOCAL_FORMAT

REF_DELETE_MARKER_TAG_BYTES

private static final byte[] REF_DELETE_MARKER_TAG_BYTES

Constructor Detail

MobUtils

private MobUtils()

Private constructor to keep this class from being instantiated.

Method Detail

formatDate

public static String formatDate(Date date)

Formats a date to a string.

Parameters:

date - The date.

Returns:

The string format of the date, it's yyyymmdd.

parseDate

public static Date parseDate(String dateString)
                      throws ParseException

Parses the string to a date.

Parameters:

dateString - The string format of a date, it's yyyymmdd.

Returns:

A date.

Throws:

ParseException

getFirstDayOfMonth

public static Date getFirstDayOfMonth(Calendar calendar,
                                      Date date)

Get the first day of the input date's month

Parameters:

calendar - Calendar object

date - The date to find out its first day of that month

Returns:

The first day in the month

getFirstDayOfWeek

public static Date getFirstDayOfWeek(Calendar calendar,
                                     Date date)

Get the first day of the input date's week

Parameters:

calendar - Calendar object

date - The date to find out its first day of that week

Returns:

The first day in the week

isMobReferenceCell

public static boolean isMobReferenceCell(Cell cell)

Whether the current cell is a mob reference cell.

Parameters:

cell - The current cell.

Returns:

True if the cell has a mob reference tag, false if it doesn't.

getTableNameTag

public static Tag getTableNameTag(Cell cell)

Gets the table name tag.

Parameters:

cell - The current cell.

Returns:

The table name tag.

hasMobReferenceTag

public static boolean hasMobReferenceTag(List<Tag> tags)

Whether the tag list has a mob reference tag.

Parameters:

tags - The tag list.

Returns:

True if the list has a mob reference tag, false if it doesn't.

isRawMobScan

public static boolean isRawMobScan(Scan scan)

Indicates whether it's a raw scan. The information is set in the attribute "hbase.mob.scan.raw" of scan. For a mob cell, in a normal scan the scanners retrieves the mob cell from the mob file. In a raw scan, the scanner directly returns cell in HBase without retrieve the one in the mob file.

Parameters:

scan - The current scan.

Returns:

True if it's a raw scan.

isRefOnlyScan

public static boolean isRefOnlyScan(Scan scan)

Indicates whether it's a reference only scan. The information is set in the attribute "hbase.mob.scan.ref.only" of scan. If it's a ref only scan, only the cells with ref tag are returned.

Parameters:

scan - The current scan.

Returns:

True if it's a ref only scan.

isCacheMobBlocks

public static boolean isCacheMobBlocks(Scan scan)

Indicates whether the scan contains the information of caching blocks. The information is set in the attribute "hbase.mob.cache.blocks" of scan.

Parameters:

scan - The current scan.

Returns:

True when the Scan attribute specifies to cache the MOB blocks.

setCacheMobBlocks

public static void setCacheMobBlocks(Scan scan,
                                     boolean cacheBlocks)

Sets the attribute of caching blocks in the scan.

Parameters:

scan - The current scan.

cacheBlocks - True, set the attribute of caching blocks into the scan, the scanner with this scan caches blocks. False, the scanner doesn't cache blocks for this scan.

cleanExpiredMobFiles

public static void cleanExpiredMobFiles(org.apache.hadoop.fs.FileSystem fs,
                                        org.apache.hadoop.conf.Configuration conf,
                                        TableName tableName,
                                        ColumnFamilyDescriptor columnDescriptor,
                                        CacheConfig cacheConfig,
                                        long current)
                                 throws IOException

Cleans the expired mob files. Cleans the files whose creation date is older than (current - columnFamily.ttl), and the minVersions of that column family is 0.

Parameters:

fs - The current file system.

conf - The current configuration.

tableName - The current table name.

columnDescriptor - The descriptor of the current column family.

cacheConfig - The cacheConfig that disables the block cache.

current - The current time.

Throws:

IOException

getMobHome

public static org.apache.hadoop.fs.Path getMobHome(org.apache.hadoop.conf.Configuration conf)

Gets the root dir of the mob files. It's {HBASE_DIR}/mobdir.

Parameters:

conf - The current configuration.

Returns:

the root dir of the mob file.

getMobHome

public static org.apache.hadoop.fs.Path getMobHome(org.apache.hadoop.fs.Path rootDir)

Gets the root dir of the mob files under the qualified HBase root dir. It's {rootDir}/mobdir.

Parameters:

rootDir - The qualified path of HBase root directory.

Returns:

The root dir of the mob file.

getQualifiedMobRootDir

public static org.apache.hadoop.fs.Path getQualifiedMobRootDir(org.apache.hadoop.conf.Configuration conf)
                                                        throws IOException

Gets the qualified root dir of the mob files.

Parameters:

conf - The current configuration.

Returns:

The qualified root dir.

Throws:

IOException

getMobTableDir

public static org.apache.hadoop.fs.Path getMobTableDir(org.apache.hadoop.fs.Path rootDir,
                                                       TableName tableName)

Gets the table dir of the mob files under the qualified HBase root dir. It's {rootDir}/mobdir/data/${namespace}/${tableName}

Parameters:

rootDir - The qualified path of HBase root directory.

tableName - The name of table.

Returns:

The table dir of the mob file.

getMobRegionPath

public static org.apache.hadoop.fs.Path getMobRegionPath(org.apache.hadoop.conf.Configuration conf,
                                                         TableName tableName)

Gets the region dir of the mob files. It's {HBASE_DIR}/mobdir/data/{namespace}/{tableName}/{regionEncodedName}.

Parameters:

conf - The current configuration.

tableName - The current table name.

Returns:

The region dir of the mob files.

getMobRegionPath

public static org.apache.hadoop.fs.Path getMobRegionPath(org.apache.hadoop.fs.Path rootDir,
                                                         TableName tableName)

Gets the region dir of the mob files under the specified root dir. It's {rootDir}/mobdir/data/{namespace}/{tableName}/{regionEncodedName}.

Parameters:

rootDir - The qualified path of HBase root directory.

tableName - The current table name.

Returns:

The region dir of the mob files.

getMobFamilyPath

public static org.apache.hadoop.fs.Path getMobFamilyPath(org.apache.hadoop.conf.Configuration conf,
                                                         TableName tableName,
                                                         String familyName)

Gets the family dir of the mob files. It's {HBASE_DIR}/mobdir/{namespace}/{tableName}/{regionEncodedName}/{columnFamilyName}.

Parameters:

conf - The current configuration.

tableName - The current table name.

familyName - The current family name.

Returns:

The family dir of the mob files.

getMobFamilyPath

public static org.apache.hadoop.fs.Path getMobFamilyPath(org.apache.hadoop.fs.Path regionPath,
                                                         String familyName)

Gets the family dir of the mob files. It's {HBASE_DIR}/mobdir/{namespace}/{tableName}/{regionEncodedName}/{columnFamilyName}.

Parameters:

regionPath - The path of mob region which is a dummy one.

familyName - The current family name.

Returns:

The family dir of the mob files.

getMobRegionInfo

public static RegionInfo getMobRegionInfo(TableName tableName)

Gets the RegionInfo of the mob files. This is a dummy region. The mob files are not saved in a region in HBase. This is only used in mob snapshot. It's internally used only.

Parameters:

tableName -

Returns:

A dummy mob region info.

isMobRegionInfo

public static boolean isMobRegionInfo(RegionInfo regionInfo)

Gets whether the current RegionInfo is a mob one.

Parameters:

regionInfo - The current RegionInfo.

Returns:

If true, the current RegionInfo is a mob one.

isMobRegionName

public static boolean isMobRegionName(TableName tableName,
                                      byte[] regionName)

Gets whether the current region name follows the pattern of a mob region name.

Parameters:

tableName - The current table name.

regionName - The current region name.

Returns:

True if the current region name follows the pattern of a mob region name.

getCompactionWorkingPath

public static org.apache.hadoop.fs.Path getCompactionWorkingPath(org.apache.hadoop.fs.Path root,
                                                                 String jobName)

Gets the working directory of the mob compaction.

Parameters:

root - The root directory of the mob compaction.

jobName - The current job name.

Returns:

The directory of the mob compaction for the current job.

removeMobFiles

public static void removeMobFiles(org.apache.hadoop.conf.Configuration conf,
                                  org.apache.hadoop.fs.FileSystem fs,
                                  TableName tableName,
                                  org.apache.hadoop.fs.Path tableDir,
                                  byte[] family,
                                  Collection<HStoreFile> storeFiles)
                           throws IOException

Archives the mob files.

Parameters:

conf - The current configuration.

fs - The current file system.

tableName - The table name.

tableDir - The table directory.

family - The name of the column family.

storeFiles - The files to be deleted.

Throws:

IOException

createMobRefCell

public static Cell createMobRefCell(Cell cell,
                                    byte[] fileName,
                                    Tag tableNameTag)

Creates a mob reference KeyValue. The value of the mob reference KeyValue is mobCellValueSize + mobFileName.

Parameters:

cell - The original Cell.

fileName - The mob file name where the mob reference KeyValue is written.

tableNameTag - The tag of the current table name. It's very important in cloning the snapshot.

Returns:

The mob reference KeyValue.

createMobRefCell

public static Cell createMobRefCell(Cell cell,
                                    byte[] fileName,
                                    byte[] refCellTags)

createWriter

public static StoreFileWriter createWriter(org.apache.hadoop.conf.Configuration conf,
                                           org.apache.hadoop.fs.FileSystem fs,
                                           ColumnFamilyDescriptor family,
                                           String date,
                                           org.apache.hadoop.fs.Path basePath,
                                           long maxKeyCount,
                                           Compression.Algorithm compression,
                                           String startKey,
                                           CacheConfig cacheConfig,
                                           Encryption.Context cryptoContext,
                                           boolean isCompaction)
                                    throws IOException

Creates a writer for the mob file in temp directory.

Parameters:

conf - The current configuration.

fs - The current file system.

family - The descriptor of the current column family.

date - The date string, its format is yyyymmmdd.

basePath - The basic path for a temp directory.

maxKeyCount - The key count.

compression - The compression algorithm.

startKey - The hex string of the start key.

cacheConfig - The current cache config.

cryptoContext - The encryption context.

isCompaction - If the writer is used in compaction.

Returns:

The writer for the mob file.

Throws:

IOException

createRefFileWriter

public static StoreFileWriter createRefFileWriter(org.apache.hadoop.conf.Configuration conf,
                                                  org.apache.hadoop.fs.FileSystem fs,
                                                  ColumnFamilyDescriptor family,
                                                  org.apache.hadoop.fs.Path basePath,
                                                  long maxKeyCount,
                                                  CacheConfig cacheConfig,
                                                  Encryption.Context cryptoContext,
                                                  boolean isCompaction)
                                           throws IOException

Creates a writer for the ref file in temp directory.

Parameters:

conf - The current configuration.

fs - The current file system.

family - The descriptor of the current column family.

basePath - The basic path for a temp directory.

maxKeyCount - The key count.

cacheConfig - The current cache config.

cryptoContext - The encryption context.

isCompaction - If the writer is used in compaction.

Returns:

The writer for the mob file.

Throws:

IOException

createWriter

public static StoreFileWriter createWriter(org.apache.hadoop.conf.Configuration conf,
                                           org.apache.hadoop.fs.FileSystem fs,
                                           ColumnFamilyDescriptor family,
                                           String date,
                                           org.apache.hadoop.fs.Path basePath,
                                           long maxKeyCount,
                                           Compression.Algorithm compression,
                                           byte[] startKey,
                                           CacheConfig cacheConfig,
                                           Encryption.Context cryptoContext,
                                           boolean isCompaction)
                                    throws IOException

Creates a writer for the mob file in temp directory.

Parameters:

conf - The current configuration.

fs - The current file system.

family - The descriptor of the current column family.

date - The date string, its format is yyyymmmdd.

basePath - The basic path for a temp directory.

maxKeyCount - The key count.

compression - The compression algorithm.

startKey - The start key.

cacheConfig - The current cache config.

cryptoContext - The encryption context.

isCompaction - If the writer is used in compaction.

Returns:

The writer for the mob file.

Throws:

IOException

createDelFileWriter

public static StoreFileWriter createDelFileWriter(org.apache.hadoop.conf.Configuration conf,
                                                  org.apache.hadoop.fs.FileSystem fs,
                                                  ColumnFamilyDescriptor family,
                                                  String date,
                                                  org.apache.hadoop.fs.Path basePath,
                                                  long maxKeyCount,
                                                  Compression.Algorithm compression,
                                                  byte[] startKey,
                                                  CacheConfig cacheConfig,
                                                  Encryption.Context cryptoContext)
                                           throws IOException

Creates a writer for the del file in temp directory.

Parameters:

conf - The current configuration.

fs - The current file system.

family - The descriptor of the current column family.

date - The date string, its format is yyyymmmdd.

basePath - The basic path for a temp directory.

maxKeyCount - The key count.

compression - The compression algorithm.

startKey - The start key.

cacheConfig - The current cache config.

cryptoContext - The encryption context.

Returns:

The writer for the del file.

Throws:

IOException

createWriter

public static StoreFileWriter createWriter(org.apache.hadoop.conf.Configuration conf,
                                           org.apache.hadoop.fs.FileSystem fs,
                                           ColumnFamilyDescriptor family,
                                           MobFileName mobFileName,
                                           org.apache.hadoop.fs.Path basePath,
                                           long maxKeyCount,
                                           Compression.Algorithm compression,
                                           CacheConfig cacheConfig,
                                           Encryption.Context cryptoContext,
                                           boolean isCompaction)
                                    throws IOException

Creates a writer for the mob file in temp directory.

Parameters:

conf - The current configuration.

fs - The current file system.

family - The descriptor of the current column family.

mobFileName - The mob file name.

basePath - The basic path for a temp directory.

maxKeyCount - The key count.

compression - The compression algorithm.

cacheConfig - The current cache config.

cryptoContext - The encryption context.

isCompaction - If the writer is used in compaction.

Returns:

The writer for the mob file.

Throws:

IOException

createWriter

public static StoreFileWriter createWriter(org.apache.hadoop.conf.Configuration conf,
                                           org.apache.hadoop.fs.FileSystem fs,
                                           ColumnFamilyDescriptor family,
                                           org.apache.hadoop.fs.Path path,
                                           long maxKeyCount,
                                           Compression.Algorithm compression,
                                           CacheConfig cacheConfig,
                                           Encryption.Context cryptoContext,
                                           ChecksumType checksumType,
                                           int bytesPerChecksum,
                                           int blocksize,
                                           BloomType bloomType,
                                           boolean isCompaction)
                                    throws IOException

Creates a writer for the mob file in temp directory.

Parameters:

conf - The current configuration.

fs - The current file system.

family - The descriptor of the current column family.

path - The path for a temp directory.

maxKeyCount - The key count.

compression - The compression algorithm.

cacheConfig - The current cache config.

cryptoContext - The encryption context.

checksumType - The checksum type.

bytesPerChecksum - The bytes per checksum.

blocksize - The HFile block size.

bloomType - The bloom filter type.

isCompaction - If the writer is used in compaction.

Returns:

The writer for the mob file.

Throws:

IOException

commitFile

public static org.apache.hadoop.fs.Path commitFile(org.apache.hadoop.conf.Configuration conf,
                                                   org.apache.hadoop.fs.FileSystem fs,
                                                   org.apache.hadoop.fs.Path sourceFile,
                                                   org.apache.hadoop.fs.Path targetPath,
                                                   CacheConfig cacheConfig)
                                            throws IOException

Commits the mob file.

Parameters:

conf - The current configuration.

fs - The current file system.

sourceFile - The path where the mob file is saved.

targetPath - The directory path where the source file is renamed to.

cacheConfig - The current cache config.

Returns:

The target file path the source file is renamed to.

Throws:

IOException

validateMobFile

private static void validateMobFile(org.apache.hadoop.conf.Configuration conf,
                                    org.apache.hadoop.fs.FileSystem fs,
                                    org.apache.hadoop.fs.Path path,
                                    CacheConfig cacheConfig,
                                    boolean primaryReplica)
                             throws IOException

Validates a mob file by opening and closing it.

Parameters:

conf - The current configuration.

fs - The current file system.

path - The path where the mob file is saved.

cacheConfig - The current cache config.

Throws:

IOException

hasValidMobRefCellValue

public static boolean hasValidMobRefCellValue(Cell cell)

Indicates whether the current mob ref cell has a valid value. A mob ref cell has a mob reference tag. The value of a mob ref cell consists of two parts, real mob value length and mob file name. The real mob value length takes 4 bytes. The remaining part is the mob file name.

Parameters:

cell - The mob ref cell.

Returns:

True if the cell has a valid value.

getMobValueLength

public static int getMobValueLength(Cell cell)

Gets the mob value length from the mob ref cell. A mob ref cell has a mob reference tag. The value of a mob ref cell consists of two parts, real mob value length and mob file name. The real mob value length takes 4 bytes. The remaining part is the mob file name.

Parameters:

cell - The mob ref cell.

Returns:

The real mob value length.

getMobFileName

public static String getMobFileName(Cell cell)

Gets the mob file name from the mob ref cell. A mob ref cell has a mob reference tag. The value of a mob ref cell consists of two parts, real mob value length and mob file name. The real mob value length takes 4 bytes. The remaining part is the mob file name.

Parameters:

cell - The mob ref cell.

Returns:

The mob file name.

getTableLockName

public static TableName getTableLockName(TableName tn)

Gets the table name used in the table lock. The table lock name is a dummy one, it's not a table name. It's tableName + ".mobLock".

Parameters:

tn - The table name.

Returns:

The table name used in table lock.

doMobCompaction

public static void doMobCompaction(org.apache.hadoop.conf.Configuration conf,
                                   org.apache.hadoop.fs.FileSystem fs,
                                   TableName tableName,
                                   ColumnFamilyDescriptor hcd,
                                   ExecutorService pool,
                                   boolean allFiles,
                                   LockManager.MasterLock lock)
                            throws IOException

Performs the mob compaction.

Parameters:

conf - the Configuration

fs - the file system

tableName - the table the compact

hcd - the column descriptor

pool - the thread pool

allFiles - Whether add all mob files into the compaction.

Throws:

IOException

createMobCompactorThreadPool

public static ExecutorService createMobCompactorThreadPool(org.apache.hadoop.conf.Configuration conf)

Creates a thread pool.

Parameters:

conf - the Configuration

Returns:

A thread pool.

hasMobColumns

public static boolean hasMobColumns(TableDescriptor htd)

Checks whether this table has mob-enabled columns.

Parameters:

htd - The current table descriptor.

Returns:

Whether this table has mob-enabled columns.

isReadEmptyValueOnMobCellMiss

public static boolean isReadEmptyValueOnMobCellMiss(Scan scan)

Indicates whether return null value when the mob file is missing or corrupt. The information is set in the attribute "empty.value.on.mobcell.miss" of scan.

Parameters:

scan - The current scan.

Returns:

True if the readEmptyValueOnMobCellMiss is enabled.

createMobRefDeleteMarker

public static Cell createMobRefDeleteMarker(Cell cell)

Creates a mob ref delete marker.

Parameters:

cell - The current delete marker.

Returns:

A delete marker with the ref tag.

isMobFileExpired

public static boolean isMobFileExpired(ColumnFamilyDescriptor column,
                                       long current,
                                       String fileDate)

Checks if the mob file is expired.

Parameters:

column - The descriptor of the current column family.

current - The current time.

fileDate - The date string parsed from the mob file name.

Returns:

True if the mob file is expired.

fillPartitionId

public static boolean fillPartitionId(PartitionedMobCompactionRequest.CompactionPartitionId id,
                                      Date firstDayOfCurrentMonth,
                                      Date firstDayOfCurrentWeek,
                                      String dateStr,
                                      MobCompactPartitionPolicy policy,
                                      Calendar calendar,
                                      long threshold)

fill out partition id based on compaction policy and date, threshold...

Parameters:

id - Partition id to be filled out

firstDayOfCurrentMonth - The first day in the current month

firstDayOfCurrentWeek - The first day in the current week

dateStr - Date string from the mob file

policy - Mob compaction policy

calendar - Calendar object

threshold - Mob compaciton threshold configured

Returns:

true if the file needs to be excluded from compaction