org.apache.hadoop.hbase.fs

Class HFileSystem

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.fs.FileSystem

org.apache.hadoop.fs.FilterFileSystem

org.apache.hadoop.hbase.fs.HFileSystem

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.conf.Configurable

@InterfaceAudience.Private
public class HFileSystem
extends org.apache.hadoop.fs.FilterFileSystem

An encapsulation for the FileSystem object that hbase uses to access data. This class allows the flexibility of using separate filesystem objects for reading and writing hfiles and wals.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
(package private) static interface	HFileSystem.ReorderBlocks Interface to implement to add a specific reordering logic in hdfs.
(package private) static class	HFileSystem.ReorderWALBlocks We're putting at lowest priority the wal files blocks that are on the same datanode as the original regionserver which created these files.

Nested classes/interfaces inherited from class org.apache.hadoop.fs.FileSystem

org.apache.hadoop.fs.FileSystem.DirectoryEntries, org.apache.hadoop.fs.FileSystem.Statistics

Field Summary

Fields

Modifier and Type	Field and Description
static org.slf4j.Logger	LOG
private org.apache.hadoop.fs.FileSystem	noChecksumFs
private static byte	unspecifiedStoragePolicyId
private boolean	useHBaseChecksum

Fields inherited from class org.apache.hadoop.fs.FilterFileSystem

fs, swapScheme

Fields inherited from class org.apache.hadoop.fs.FileSystem

DEFAULT_FS, FS_DEFAULT_NAME_KEY, SHUTDOWN_HOOK_PRIORITY, statistics, TRASH_PREFIX, USER_HOME_PREFIX

Constructor Summary

Constructors

Constructor and Description
HFileSystem(org.apache.hadoop.conf.Configuration conf, boolean useHBaseChecksum) Create a FileSystem object for HBase regionservers.
HFileSystem(org.apache.hadoop.fs.FileSystem fs) Wrap a FileSystem object within a HFileSystem.

Method Summary

Modifier and Type	Method and Description
static boolean	addLocationsOrderInterceptor(org.apache.hadoop.conf.Configuration conf)
(package private) static boolean	addLocationsOrderInterceptor(org.apache.hadoop.conf.Configuration conf, HFileSystem.ReorderBlocks lrb) Add an interceptor on the calls to the namenode#getBlockLocations from the DFSClient linked to this FileSystem.
void	close() Close this filesystem object
org.apache.hadoop.fs.FSDataOutputStream	createNonRecursive(org.apache.hadoop.fs.Path f, boolean overwrite, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress) The org.apache.hadoop.fs.FilterFileSystem does not yet support createNonRecursive.
private static org.apache.hadoop.hdfs.protocol.ClientProtocol	createReorderingProxy(org.apache.hadoop.hdfs.protocol.ClientProtocol cp, HFileSystem.ReorderBlocks lrb, org.apache.hadoop.conf.Configuration conf)
static org.apache.hadoop.fs.FileSystem	get(org.apache.hadoop.conf.Configuration conf) Create a new HFileSystem object, similar to FileSystem.get().
org.apache.hadoop.fs.FileSystem	getBackingFs() Returns the underlying filesystem
static org.apache.hadoop.fs.FileSystem	getLocalFs(org.apache.hadoop.conf.Configuration conf) Wrap a LocalFileSystem within a HFileSystem.
org.apache.hadoop.fs.FileSystem	getNoChecksumFs() Returns the filesystem that is specially setup for doing reads from storage.
private String	getStoragePolicyForOldHDFSVersion(org.apache.hadoop.fs.Path path) Before Hadoop 2.8.0, there's no getStoragePolicy method for FileSystem interface, and we need to keep compatible with it.
String	getStoragePolicyName(org.apache.hadoop.fs.Path path) Get the storage policy of the source path (directory/file).
private org.apache.hadoop.fs.FileSystem	maybeWrapFileSystem(org.apache.hadoop.fs.FileSystem base, org.apache.hadoop.conf.Configuration conf) Returns an instance of Filesystem wrapped into the class specified in hbase.fs.wrapper property, if one is set in the configuration, returns unmodified FS instance passed in as an argument otherwise.
private static org.apache.hadoop.fs.FileSystem	newInstanceFileSystem(org.apache.hadoop.conf.Configuration conf) Returns a brand new instance of the FileSystem.
void	setStoragePolicy(org.apache.hadoop.fs.Path path, String policyName) Set the source path (directory/file) to the specified storage policy.
boolean	useHBaseChecksum() Are we verifying checksums in HBase?

Methods inherited from class org.apache.hadoop.fs.FilterFileSystem

access, append, appendFile, canonicalizeUri, checkPath, completeLocalOutput, concat, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyToLocalFile, create, create, createFile, createNonRecursive, createSnapshot, createSymlink, delete, deleteSnapshot, getAclStatus, getAllStoragePolicies, getCanonicalUri, getChildFileSystems, getConf, getDefaultBlockSize, getDefaultBlockSize, getDefaultReplication, getDefaultReplication, getFileBlockLocations, getFileChecksum, getFileChecksum, getFileLinkStatus, getFileStatus, getHomeDirectory, getInitialWorkingDirectory, getLinkTarget, getRawFileSystem, getServerDefaults, getServerDefaults, getStatus, getStoragePolicy, getTrashRoot, getTrashRoots, getUri, getUsed, getUsed, getWorkingDirectory, getXAttr, getXAttrs, getXAttrs, initialize, listCorruptFileBlocks, listLocatedStatus, listLocatedStatus, listStatus, listStatusIterator, listXAttrs, makeQualified, mkdirs, mkdirs, modifyAclEntries, msync, open, primitiveCreate, primitiveMkdir, removeAcl, removeAclEntries, removeDefaultAcl, removeXAttr, rename, rename, renameSnapshot, resolveLink, resolvePath, setAcl, setOwner, setPermission, setReplication, setTimes, setVerifyChecksum, setWorkingDirectory, setWriteChecksum, setXAttr, setXAttr, startLocalOutput, supportsSymlinks, truncate, unsetStoragePolicy

Methods inherited from class org.apache.hadoop.fs.FileSystem

addDelegationTokens, append, append, areSymlinksEnabled, cancelDeleteOnExit, clearStatistics, closeAll, closeAllForUGI, copyFromLocalFile, copyToLocalFile, copyToLocalFile, create, create, create, create, create, create, create, create, create, create, create, createNewFile, createNonRecursive, createSnapshot, delete, deleteOnExit, enableSymlinks, exists, fixRelativePart, get, get, getAllStatistics, getBlockSize, getCanonicalServiceName, getContentSummary, getDefaultPort, getDefaultUri, getDelegationToken, getFileBlockLocations, getFileSystemClass, getFSofPath, getGlobalStorageStatistics, getLength, getLocal, getName, getNamed, getQuotaUsage, getReplication, getScheme, getStatistics, getStatistics, getStatus, getStorageStatistics, globStatus, globStatus, isDirectory, isFile, listFiles, listStatus, listStatus, listStatus, listStatusBatch, mkdirs, moveFromLocalFile, moveFromLocalFile, moveToLocalFile, newInstance, newInstance, newInstance, newInstanceLocal, open, primitiveMkdir, printStatistics, processDeleteOnExit, setDefaultUri, setDefaultUri

Methods inherited from class org.apache.hadoop.conf.Configured

setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.slf4j.Logger LOG

noChecksumFs

private final org.apache.hadoop.fs.FileSystem noChecksumFs

useHBaseChecksum

private final boolean useHBaseChecksum

unspecifiedStoragePolicyId

private static volatile byte unspecifiedStoragePolicyId

Constructor Detail

HFileSystem

public HFileSystem(org.apache.hadoop.conf.Configuration conf,
                   boolean useHBaseChecksum)
            throws IOException

Create a FileSystem object for HBase regionservers.

Parameters:

conf - The configuration to be used for the filesystem

useHBaseChecksum - if true, then use checksum verfication in hbase, otherwise delegate checksum verification to the FileSystem.

Throws:

IOException

HFileSystem

public HFileSystem(org.apache.hadoop.fs.FileSystem fs)

Wrap a FileSystem object within a HFileSystem. The noChecksumFs and writefs are both set to be the same specified fs. Do not verify hbase-checksums while reading data from filesystem.

Parameters:

fs - Set the noChecksumFs and writeFs to this specified filesystem.

Method Detail

getNoChecksumFs

public org.apache.hadoop.fs.FileSystem getNoChecksumFs()

Returns the filesystem that is specially setup for doing reads from storage. This object avoids doing checksum verifications for reads.

Returns:

The FileSystem object that can be used to read data from files.

getBackingFs

public org.apache.hadoop.fs.FileSystem getBackingFs()
                                             throws IOException

Returns the underlying filesystem

Returns:

The underlying FileSystem for this FilterFileSystem object.

Throws:

IOException

setStoragePolicy

public void setStoragePolicy(org.apache.hadoop.fs.Path path,
                             String policyName)

Set the source path (directory/file) to the specified storage policy.

Overrides:

setStoragePolicy in class org.apache.hadoop.fs.FilterFileSystem

Parameters:

path - The source path (directory/file).

policyName - The name of the storage policy: 'HOT', 'COLD', etc. See see hadoop 2.6+ org.apache.hadoop.hdfs.protocol.HdfsConstants for possible list e.g 'COLD', 'WARM', 'HOT', 'ONE_SSD', 'ALL_SSD', 'LAZY_PERSIST'.

getStoragePolicyName

@Nullable
public String getStoragePolicyName(org.apache.hadoop.fs.Path path)

Get the storage policy of the source path (directory/file).

Parameters:

path - The source path (directory/file).

Returns:

Storage policy name, or null if not using DistributedFileSystem or exception thrown when trying to get policy

getStoragePolicyForOldHDFSVersion

private String getStoragePolicyForOldHDFSVersion(org.apache.hadoop.fs.Path path)

Before Hadoop 2.8.0, there's no getStoragePolicy method for FileSystem interface, and we need to keep compatible with it. See HADOOP-12161 for more details.

Parameters:

path - Path to get storage policy against

Returns:

the storage policy name

useHBaseChecksum

public boolean useHBaseChecksum()

Are we verifying checksums in HBase?

Returns:

True, if hbase is configured to verify checksums, otherwise false.

close

public void close()
           throws IOException

Close this filesystem object

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Overrides:

close in class org.apache.hadoop.fs.FilterFileSystem

Throws:

IOException

newInstanceFileSystem

private static org.apache.hadoop.fs.FileSystem newInstanceFileSystem(org.apache.hadoop.conf.Configuration conf)
                                                              throws IOException

Returns a brand new instance of the FileSystem. It does not use the FileSystem.Cache. In newer versions of HDFS, we can directly invoke FileSystem.newInstance(Configuration).

Parameters:

conf - Configuration

Returns:

A new instance of the filesystem

Throws:

IOException

maybeWrapFileSystem

private org.apache.hadoop.fs.FileSystem maybeWrapFileSystem(org.apache.hadoop.fs.FileSystem base,
                                                            org.apache.hadoop.conf.Configuration conf)

Returns an instance of Filesystem wrapped into the class specified in hbase.fs.wrapper property, if one is set in the configuration, returns unmodified FS instance passed in as an argument otherwise.

Parameters:

base - Filesystem instance to wrap

conf - Configuration

Returns:

wrapped instance of FS, or the same instance if no wrapping configured.

addLocationsOrderInterceptor

public static boolean addLocationsOrderInterceptor(org.apache.hadoop.conf.Configuration conf)
                                            throws IOException

Throws:

IOException

addLocationsOrderInterceptor

static boolean addLocationsOrderInterceptor(org.apache.hadoop.conf.Configuration conf,
                                            HFileSystem.ReorderBlocks lrb)

Add an interceptor on the calls to the namenode#getBlockLocations from the DFSClient linked to this FileSystem. See HBASE-6435 for the background.

There should be no reason, except testing, to create a specific ReorderBlocks.

Returns:

true if the interceptor was added, false otherwise.

createReorderingProxy

private static org.apache.hadoop.hdfs.protocol.ClientProtocol createReorderingProxy(org.apache.hadoop.hdfs.protocol.ClientProtocol cp,
                                                                                    HFileSystem.ReorderBlocks lrb,
                                                                                    org.apache.hadoop.conf.Configuration conf)

get

public static org.apache.hadoop.fs.FileSystem get(org.apache.hadoop.conf.Configuration conf)
                                           throws IOException

Create a new HFileSystem object, similar to FileSystem.get(). This returns a filesystem object that avoids checksum verification in the filesystem for hfileblock-reads. For these blocks, checksum verification is done by HBase.

Throws:

IOException

getLocalFs

public static org.apache.hadoop.fs.FileSystem getLocalFs(org.apache.hadoop.conf.Configuration conf)
                                                  throws IOException

Wrap a LocalFileSystem within a HFileSystem.

Throws:

IOException

createNonRecursive

public org.apache.hadoop.fs.FSDataOutputStream createNonRecursive(org.apache.hadoop.fs.Path f,
                                                                  boolean overwrite,
                                                                  int bufferSize,
                                                                  short replication,
                                                                  long blockSize,
                                                                  org.apache.hadoop.util.Progressable progress)
                                                           throws IOException

The org.apache.hadoop.fs.FilterFileSystem does not yet support createNonRecursive. This is a hadoop bug and when it is fixed in Hadoop, this definition will go away.

Overrides:

createNonRecursive in class org.apache.hadoop.fs.FileSystem

Throws:

IOException