org.apache.hadoop.hbase.util

Class FSTableDescriptors

java.lang.Object

org.apache.hadoop.hbase.util.FSTableDescriptors

All Implemented Interfaces:

TableDescriptors

@InterfaceAudience.Private
public class FSTableDescriptors
extends Object
implements TableDescriptors

Implementation of TableDescriptors that reads descriptors from the passed filesystem. It expects descriptors to be in a file in the TABLEINFO_DIR subdir of the table's directory in FS. Can be read-only -- i.e. does not modify the filesystem or can be read and write.

Also has utility for keeping up the table descriptors tableinfo file. The table schema file is kept in the TABLEINFO_DIR subdir of the table directory in the filesystem. It has a TABLEINFO_FILE_PREFIX and then a suffix that is the edit sequenceid: e.g. .tableinfo.0000000003. This sequenceid is always increasing. It starts at zero. The table schema file with the highest sequenceid has the most recent schema edit. Usually there is one file only, the most recent but there may be short periods where there are more than one file. Old files are eventually cleaned. Presumption is that there will not be lots of concurrent clients making table schema edits. If so, the below needs a bit of a reworking and perhaps some supporting api in hdfs.

Field Summary

Fields

Modifier and Type	Field and Description
private Map<TableName,TableDescriptor>	cache
(package private) long	cachehits
private org.apache.hadoop.fs.FileSystem	fs
private boolean	fsreadonly
private boolean	fsvisited
(package private) long	invocations
private static org.slf4j.Logger	LOG
private org.apache.hadoop.fs.Path	rootdir
(package private) static String	TABLEINFO_DIR
(package private) static String	TABLEINFO_FILE_PREFIX The file name prefix used to store HTD in HDFS
private static Pattern	TABLEINFO_FILE_REGEX Regex to eat up sequenceid suffix on a .tableinfo file.
(package private) static Comparator<org.apache.hadoop.fs.FileStatus>	TABLEINFO_FILESTATUS_COMPARATOR Compare FileStatus instances by Path.getName().
private static org.apache.hadoop.fs.PathFilter	TABLEINFO_PATHFILTER
(package private) static String	TMP_DIR
private boolean	usecache
(package private) static int	WIDTH_OF_SEQUENCE_ID Width of the sequenceid that is a suffix on a tableinfo file.

Constructor Summary

Constructors

Constructor and Description
FSTableDescriptors(org.apache.hadoop.conf.Configuration conf) Construct a FSTableDescriptors instance using the hbase root dir of the given conf and the filesystem where that root dir lives.
FSTableDescriptors(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path rootdir)
FSTableDescriptors(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path rootdir, boolean fsreadonly, boolean usecache)

Method Summary

Modifier and Type	Method and Description
static TableDescriptorBuilder	createMetaTableDescriptorBuilder(org.apache.hadoop.conf.Configuration conf)
boolean	createTableDescriptor(TableDescriptor htd) Create new TableDescriptor in HDFS.
boolean	createTableDescriptor(TableDescriptor htd, boolean forceCreation) Create new TableDescriptor in HDFS.
static boolean	createTableDescriptorForTableDirectory(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path tableDir, TableDescriptor htd, boolean forceCreation) Create a new TableDescriptor in HDFS in the specified table directory.
boolean	createTableDescriptorForTableDirectory(org.apache.hadoop.fs.Path tableDir, TableDescriptor htd, boolean forceCreation) Create a new TableDescriptor in HDFS in the specified table directory.
private static void	deleteTableDescriptorFiles(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path dir, int maxSequenceId) Deletes files matching the table info file pattern within the given directory whose sequenceId is at most the given max sequenceId.
private static String	formatTableInfoSequenceId(int number)
TableDescriptor	get(TableName tableName) Get the current table descriptor for the given table, or null if none exists.
Map<String,TableDescriptor>	getAll() Returns a map from table name to table descriptor for all tables.
Map<String,TableDescriptor>	getByNamespace(String name) Find descriptors by namespace.
private static org.apache.hadoop.fs.FileStatus	getCurrentTableInfoStatus(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path dir, boolean removeOldFiles) Find the most current table info file in the given directory Looks within the given directory for any table info files and takes the 'current' one - meaning the one with the highest sequence number if present or no sequence number at all if none exist (for backward compatibility from before there were sequence numbers).
static TableDescriptor	getTableDescriptorFromFs(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path tableDir) Returns the latest table descriptor for the table located at the given directory directly from the file system if it exists.
static TableDescriptor	getTableDescriptorFromFs(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path hbaseRootDir, TableName tableName) Returns the latest table descriptor for the given table directly from the file system if it exists, bypassing the local cache.
(package private) org.apache.hadoop.fs.Path	getTableDir(TableName tableName) Return the table directory in HDFS
(package private) static String	getTableInfoFileName(int sequenceid)
static org.apache.hadoop.fs.FileStatus	getTableInfoPath(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path tableDir) Find the most current table info file for the table located in the given table directory.
private static org.apache.hadoop.fs.FileStatus	getTableInfoPath(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path tableDir, boolean removeOldFiles) Find the most current table info file for the table in the given table directory.
private org.apache.hadoop.fs.FileStatus	getTableInfoPath(org.apache.hadoop.fs.Path tableDir)
(package private) static int	getTableInfoSequenceId(org.apache.hadoop.fs.Path p)
protected boolean	isUsecache()
private static TableDescriptor	readTableDescriptor(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.FileStatus status)
TableDescriptor	remove(TableName tablename) Removes the table descriptor from the local cache and returns it.
static TableDescriptor	tryUpdateAndGetMetaTableDescriptor(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path rootdir)
static void	tryUpdateMetaTableDescriptor(org.apache.hadoop.conf.Configuration conf)
void	update(TableDescriptor td, boolean cacheOnly) Add or update descriptor
(package private) org.apache.hadoop.fs.Path	updateTableDescriptor(TableDescriptor td)
private static org.apache.hadoop.fs.Path	writeTableDescriptor(org.apache.hadoop.fs.FileSystem fs, TableDescriptor htd, org.apache.hadoop.fs.Path tableDir, org.apache.hadoop.fs.FileStatus currentDescriptorFile) Attempts to write a new table descriptor to the given table's directory.
private static void	writeTD(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path p, TableDescriptor htd)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.hbase.TableDescriptors

exists, update

Field Detail

LOG

private static final org.slf4j.Logger LOG

fs

private final org.apache.hadoop.fs.FileSystem fs

rootdir

private final org.apache.hadoop.fs.Path rootdir

fsreadonly

private final boolean fsreadonly

usecache

private final boolean usecache

fsvisited

private volatile boolean fsvisited

cachehits

long cachehits

invocations

long invocations

TABLEINFO_FILE_PREFIX

static final String TABLEINFO_FILE_PREFIX

The file name prefix used to store HTD in HDFS

See Also:

Constant Field Values

TABLEINFO_DIR

static final String TABLEINFO_DIR

See Also:

Constant Field Values

TMP_DIR

static final String TMP_DIR

See Also:

Constant Field Values

cache

private final Map<TableName,TableDescriptor> cache

TABLEINFO_FILESTATUS_COMPARATOR

static final Comparator<org.apache.hadoop.fs.FileStatus> TABLEINFO_FILESTATUS_COMPARATOR

Compare FileStatus instances by Path.getName(). Returns in reverse order.

TABLEINFO_PATHFILTER

private static final org.apache.hadoop.fs.PathFilter TABLEINFO_PATHFILTER

WIDTH_OF_SEQUENCE_ID

static final int WIDTH_OF_SEQUENCE_ID

Width of the sequenceid that is a suffix on a tableinfo file.

See Also:

Constant Field Values

TABLEINFO_FILE_REGEX

private static final Pattern TABLEINFO_FILE_REGEX

Regex to eat up sequenceid suffix on a .tableinfo file. Use regex because may encounter oldstyle .tableinfos where there is no sequenceid on the end.

Constructor Detail

FSTableDescriptors

public FSTableDescriptors(org.apache.hadoop.conf.Configuration conf)
                   throws IOException

Construct a FSTableDescriptors instance using the hbase root dir of the given conf and the filesystem where that root dir lives. This instance can do write operations (is not read only).

Throws:

IOException

FSTableDescriptors

public FSTableDescriptors(org.apache.hadoop.fs.FileSystem fs,
                          org.apache.hadoop.fs.Path rootdir)

FSTableDescriptors

public FSTableDescriptors(org.apache.hadoop.fs.FileSystem fs,
                          org.apache.hadoop.fs.Path rootdir,
                          boolean fsreadonly,
                          boolean usecache)

Method Detail

tryUpdateMetaTableDescriptor

public static void tryUpdateMetaTableDescriptor(org.apache.hadoop.conf.Configuration conf)
                                         throws IOException

Throws:

IOException

tryUpdateAndGetMetaTableDescriptor

public static TableDescriptor tryUpdateAndGetMetaTableDescriptor(org.apache.hadoop.conf.Configuration conf,
                                                                 org.apache.hadoop.fs.FileSystem fs,
                                                                 org.apache.hadoop.fs.Path rootdir)
                                                          throws IOException

Throws:

IOException

createMetaTableDescriptorBuilder

public static TableDescriptorBuilder createMetaTableDescriptorBuilder(org.apache.hadoop.conf.Configuration conf)
                                                               throws IOException

Throws:

IOException

isUsecache

protected boolean isUsecache()

get

@Nullable
public TableDescriptor get(TableName tableName)

Get the current table descriptor for the given table, or null if none exists.

Uses a local cache of the descriptor but still checks the filesystem on each call if fsvisited is not true, i.e, we haven't done a full scan yet, to see if a newer file has been created since the cached one was read.

Specified by:

get in interface TableDescriptors

Returns:

TableDescriptor for tablename

getAll

public Map<String,TableDescriptor> getAll()
                                   throws IOException

Returns a map from table name to table descriptor for all tables.

Specified by:

getAll in interface TableDescriptors

Returns:

Map of all descriptors.

Throws:

IOException

getByNamespace

public Map<String,TableDescriptor> getByNamespace(String name)
                                           throws IOException

Find descriptors by namespace.

Specified by:

getByNamespace in interface TableDescriptors

Returns:

Map of all descriptors.

Throws:

IOException

See Also:

get(org.apache.hadoop.hbase.TableName)

update

public void update(TableDescriptor td,
                   boolean cacheOnly)
            throws IOException

Description copied from interface: TableDescriptors

Add or update descriptor

Specified by:

update in interface TableDescriptors

Parameters:

td - Descriptor to set into TableDescriptors

cacheOnly - only add the given htd to cache, without updating the storage. For example, when creating table, we will write the descriptor to fs when creating the fs layout, so we do not need to update the fs again.

Throws:

IOException

updateTableDescriptor

org.apache.hadoop.fs.Path updateTableDescriptor(TableDescriptor td)
                                         throws IOException

Throws:

IOException

remove

public TableDescriptor remove(TableName tablename)
                       throws IOException

Removes the table descriptor from the local cache and returns it. If not in read only mode, it also deletes the entire table directory(!) from the FileSystem.

Specified by:

remove in interface TableDescriptors

Returns:

Instance of table descriptor or null if none found.

Throws:

IOException

getTableInfoPath

private org.apache.hadoop.fs.FileStatus getTableInfoPath(org.apache.hadoop.fs.Path tableDir)
                                                  throws IOException

Throws:

IOException

getTableInfoPath

public static org.apache.hadoop.fs.FileStatus getTableInfoPath(org.apache.hadoop.fs.FileSystem fs,
                                                               org.apache.hadoop.fs.Path tableDir)
                                                        throws IOException

Find the most current table info file for the table located in the given table directory. Looks within the TABLEINFO_DIR subdirectory of the given directory for any table info files and takes the 'current' one - meaning the one with the highest sequence number if present or no sequence number at all if none exist (for backward compatibility from before there were sequence numbers).

Returns:

The file status of the current table info file or null if it does not exist

Throws:

IOException

getTableInfoPath

private static org.apache.hadoop.fs.FileStatus getTableInfoPath(org.apache.hadoop.fs.FileSystem fs,
                                                                org.apache.hadoop.fs.Path tableDir,
                                                                boolean removeOldFiles)
                                                         throws IOException

Find the most current table info file for the table in the given table directory. Looks within the TABLEINFO_DIR subdirectory of the given directory for any table info files and takes the 'current' one - meaning the one with the highest sequence number if present or no sequence number at all if none exist (for backward compatibility from before there were sequence numbers). If there are multiple table info files found and removeOldFiles is true it also deletes the older files.

Returns:

The file status of the current table info file or null if none exist

Throws:

IOException

getCurrentTableInfoStatus

private static org.apache.hadoop.fs.FileStatus getCurrentTableInfoStatus(org.apache.hadoop.fs.FileSystem fs,
                                                                         org.apache.hadoop.fs.Path dir,
                                                                         boolean removeOldFiles)
                                                                  throws IOException

Find the most current table info file in the given directory

Looks within the given directory for any table info files and takes the 'current' one - meaning the one with the highest sequence number if present or no sequence number at all if none exist (for backward compatibility from before there were sequence numbers).

If there are multiple possible files found and the we're not in read only mode it also deletes the older files.

Returns:

The file status of the current table info file or null if it does not exist

Throws:

IOException

getTableDir

org.apache.hadoop.fs.Path getTableDir(TableName tableName)

Return the table directory in HDFS

formatTableInfoSequenceId

private static String formatTableInfoSequenceId(int number)

Parameters:

number - Number to use as suffix.

Returns:

Returns zero-prefixed decimal version of passed number (Does absolute in case number is negative).

getTableInfoSequenceId

static int getTableInfoSequenceId(org.apache.hadoop.fs.Path p)

Parameters:

p - Path to a .tableinfo file.

Returns:

The current editid or 0 if none found.

getTableInfoFileName

static String getTableInfoFileName(int sequenceid)

Parameters:

sequenceid -

Returns:

Name of tableinfo file.

getTableDescriptorFromFs

public static TableDescriptor getTableDescriptorFromFs(org.apache.hadoop.fs.FileSystem fs,
                                                       org.apache.hadoop.fs.Path hbaseRootDir,
                                                       TableName tableName)
                                                throws IOException

Returns the latest table descriptor for the given table directly from the file system if it exists, bypassing the local cache. Returns null if it's not found.

Throws:

IOException

getTableDescriptorFromFs

public static TableDescriptor getTableDescriptorFromFs(org.apache.hadoop.fs.FileSystem fs,
                                                       org.apache.hadoop.fs.Path tableDir)
                                                throws IOException

Returns the latest table descriptor for the table located at the given directory directly from the file system if it exists.

Throws:

TableInfoMissingException - if there is no descriptor

IOException

readTableDescriptor

private static TableDescriptor readTableDescriptor(org.apache.hadoop.fs.FileSystem fs,
                                                   org.apache.hadoop.fs.FileStatus status)
                                            throws IOException

Throws:

IOException

deleteTableDescriptorFiles

private static void deleteTableDescriptorFiles(org.apache.hadoop.fs.FileSystem fs,
                                               org.apache.hadoop.fs.Path dir,
                                               int maxSequenceId)
                                        throws IOException

Deletes files matching the table info file pattern within the given directory whose sequenceId is at most the given max sequenceId.

Throws:

IOException

writeTableDescriptor

private static org.apache.hadoop.fs.Path writeTableDescriptor(org.apache.hadoop.fs.FileSystem fs,
                                                              TableDescriptor htd,
                                                              org.apache.hadoop.fs.Path tableDir,
                                                              org.apache.hadoop.fs.FileStatus currentDescriptorFile)
                                                       throws IOException

Attempts to write a new table descriptor to the given table's directory. It first writes it to the .tmp dir then uses an atomic rename to move it into place. It begins at the currentSequenceId + 1 and tries 10 times to find a new sequence number not already in use.

Removes the current descriptor file if passed in.

Returns:

Descriptor file or null if we failed write.

Throws:

IOException

writeTD

private static void writeTD(org.apache.hadoop.fs.FileSystem fs,
                            org.apache.hadoop.fs.Path p,
                            TableDescriptor htd)
                     throws IOException

Throws:

IOException

createTableDescriptor

public boolean createTableDescriptor(TableDescriptor htd)
                              throws IOException

Create new TableDescriptor in HDFS. Happens when we are creating table. Used by tests.

Returns:

True if we successfully created file.

Throws:

IOException

createTableDescriptor

public boolean createTableDescriptor(TableDescriptor htd,
                                     boolean forceCreation)
                              throws IOException

Create new TableDescriptor in HDFS. Happens when we are creating table. If forceCreation is true then even if previous table descriptor is present it will be overwritten

Returns:

True if we successfully created file.

Throws:

IOException

createTableDescriptorForTableDirectory

public boolean createTableDescriptorForTableDirectory(org.apache.hadoop.fs.Path tableDir,
                                                      TableDescriptor htd,
                                                      boolean forceCreation)
                                               throws IOException

Create a new TableDescriptor in HDFS in the specified table directory. Happens when we create a new table during cluster start or in Clone and Create Table Procedures. Checks readOnly flag passed on construction.

Parameters:

tableDir - table directory under which we should write the file

htd - description of the table to write

forceCreation - if true,then even if previous table descriptor is present it will be overwritten

Returns:

true if the we successfully created the file, false if the file already exists and we weren't forcing the descriptor creation.

Throws:

IOException - if a filesystem error occurs

createTableDescriptorForTableDirectory

public static boolean createTableDescriptorForTableDirectory(org.apache.hadoop.fs.FileSystem fs,
                                                             org.apache.hadoop.fs.Path tableDir,
                                                             TableDescriptor htd,
                                                             boolean forceCreation)
                                                      throws IOException

Create a new TableDescriptor in HDFS in the specified table directory. Happens when we create a new table snapshoting. Does not enforce read-only. That is for caller to determine.

Parameters:

fs - Filesystem to use.

tableDir - table directory under which we should write the file

htd - description of the table to write

forceCreation - if true,then even if previous table descriptor is present it will be overwritten

Returns:

true if the we successfully created the file, false if the file already exists and we weren't forcing the descriptor creation.

Throws:

IOException - if a filesystem error occurs