org.apache.hadoop.hbase.io

Class FileLink

java.lang.Object

org.apache.hadoop.hbase.io.FileLink

Direct Known Subclasses:

HFileLink, WALLink

@InterfaceAudience.Private
public class FileLink
extends Object

The FileLink is a sort of hardlink, that allows access to a file given a set of locations.

The Problem:

• HDFS doesn't have support for hardlinks, and this make impossible to referencing the same data blocks using different names.
• HBase store files in one location (e.g. table/region/family/) and when the file is not needed anymore (e.g. compaction, region deletion, ...) moves it to an archive directory.

If we want to create a reference to a file, we need to remember that it can be in its original location or in the archive folder. The FileLink class tries to abstract this concept and given a set of locations it is able to switch between them making this operation transparent for the user. HFileLink is a more concrete implementation of the FileLink.

Back-references: To help the CleanerChore to keep track of the links to a particular file, during the FileLink creation, a new file is placed inside a back-reference directory. There's one back-reference directory for each file that has links, and in the directory there's one file per link.

HFileLink Example

• /hbase/table/region-x/cf/file-k (Original File)
• /hbase/table-cloned/region-y/cf/file-k.region-x.table (HFileLink to the original file)
• /hbase/table-2nd-cloned/region-z/cf/file-k.region-x.table (HFileLink to the original file)
• /hbase/.archive/table/region-x/.links-file-k/region-y.table-cloned (Back-reference to the link in table-cloned)
• /hbase/.archive/table/region-x/.links-file-k/region-z.table-2nd-cloned (Back-reference to the link in table-2nd-cloned)

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
private static class	FileLink.FileLinkInputStream FileLink InputStream that handles the switch between the original path and the alternative locations, when the file is moved.

Field Summary

Fields

Modifier and Type	Field and Description
static String	BACK_REFERENCES_DIRECTORY_PREFIX Define the Back-reference directory name prefix: .links-<hfile>/
private org.apache.hadoop.fs.Path[]	locations
private static org.slf4j.Logger	LOG

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	FileLink()
	FileLink(Collection<org.apache.hadoop.fs.Path> locations)
	FileLink(org.apache.hadoop.fs.Path originPath, org.apache.hadoop.fs.Path... alternativePaths)

Method Summary

Modifier and Type	Method and Description
boolean	equals(Object obj)
boolean	exists(org.apache.hadoop.fs.FileSystem fs) Returns true if the file pointed by the link exists
org.apache.hadoop.fs.Path	getAvailablePath(org.apache.hadoop.fs.FileSystem fs) Returns the path of the first available link.
static String	getBackReferenceFileName(org.apache.hadoop.fs.Path dirPath) Get the referenced file name from the reference link directory path.
static org.apache.hadoop.fs.Path	getBackReferencesDir(org.apache.hadoop.fs.Path storeDir, String fileName) Get the directory to store the link back references
org.apache.hadoop.fs.FileStatus	getFileStatus(org.apache.hadoop.fs.FileSystem fs) Get the FileStatus of the referenced file.
org.apache.hadoop.fs.Path[]	getLocations() Returns the locations to look for the linked file.
static org.apache.hadoop.fs.FSDataInputStream	getUnderlyingFileLinkInputStream(org.apache.hadoop.fs.FSDataInputStream stream) If the passed FSDataInputStream is backed by a FileLink, returns the underlying InputStream for the resolved link target.
private static IOException	handleAccessLocationException(FileLink fileLink, IOException newException, IOException previousException) Handle exceptions which are thrown when access locations of file link
int	hashCode()
static boolean	isBackReferencesDir(org.apache.hadoop.fs.Path dirPath) Checks if the specified directory path is a back reference links folder.
org.apache.hadoop.fs.FSDataInputStream	open(org.apache.hadoop.fs.FileSystem fs) Open the FileLink for read.
org.apache.hadoop.fs.FSDataInputStream	open(org.apache.hadoop.fs.FileSystem fs, int bufferSize) Open the FileLink for read.
protected void	setLocations(org.apache.hadoop.fs.Path originPath, org.apache.hadoop.fs.Path... alternativePaths) NOTE: This method must be used only in the constructor! It creates a List with the specified locations for the link.
String	toString()

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

LOG

private static final org.slf4j.Logger LOG

BACK_REFERENCES_DIRECTORY_PREFIX

public static final String BACK_REFERENCES_DIRECTORY_PREFIX

Define the Back-reference directory name prefix: .links-<hfile>/

See Also:

Constant Field Values

locations

private org.apache.hadoop.fs.Path[] locations

Constructor Detail

FileLink

protected FileLink()

FileLink

public FileLink(org.apache.hadoop.fs.Path originPath,
                org.apache.hadoop.fs.Path... alternativePaths)

Parameters:

originPath - Original location of the file to link

alternativePaths - Alternative locations to look for the linked file

FileLink

public FileLink(Collection<org.apache.hadoop.fs.Path> locations)

Parameters:

locations - locations to look for the linked file

Method Detail

getLocations

public org.apache.hadoop.fs.Path[] getLocations()

Returns the locations to look for the linked file.

toString

public String toString()

Overrides:

toString in class Object

exists

public boolean exists(org.apache.hadoop.fs.FileSystem fs)
               throws IOException

Returns true if the file pointed by the link exists

Throws:

IOException

getAvailablePath

public org.apache.hadoop.fs.Path getAvailablePath(org.apache.hadoop.fs.FileSystem fs)
                                           throws IOException

Returns the path of the first available link.

Throws:

IOException

getFileStatus

public org.apache.hadoop.fs.FileStatus getFileStatus(org.apache.hadoop.fs.FileSystem fs)
                                              throws IOException

Get the FileStatus of the referenced file.

Parameters:

fs - FileSystem on which to get the file status

Returns:

InputStream for the hfile link.

Throws:

IOException - on unexpected error.

handleAccessLocationException

private static IOException handleAccessLocationException(FileLink fileLink,
                                                         IOException newException,
                                                         IOException previousException)
                                                  throws IOException

Handle exceptions which are thrown when access locations of file link

Parameters:

fileLink - the file link

newException - the exception caught by access the current location

previousException - the previous exception caught by access the other locations

Returns:

return AccessControlException if access one of the locations caught, otherwise return FileNotFoundException. The AccessControlException is threw if user scan snapshot feature is enabled, see SnapshotScannerHDFSAclController.

Throws:

IOException - if the exception is neither AccessControlException nor FileNotFoundException

open

public org.apache.hadoop.fs.FSDataInputStream open(org.apache.hadoop.fs.FileSystem fs)
                                            throws IOException

Open the FileLink for read.

It uses a wrapper of FSDataInputStream that is agnostic to the location of the file, even if the file switches between locations.

Parameters:

fs - FileSystem on which to open the FileLink

Returns:

InputStream for reading the file link.

Throws:

IOException - on unexpected error.

open

public org.apache.hadoop.fs.FSDataInputStream open(org.apache.hadoop.fs.FileSystem fs,
                                                   int bufferSize)
                                            throws IOException

Open the FileLink for read.

It uses a wrapper of FSDataInputStream that is agnostic to the location of the file, even if the file switches between locations.

Parameters:

fs - FileSystem on which to open the FileLink

bufferSize - the size of the buffer to be used.

Returns:

InputStream for reading the file link.

Throws:

IOException - on unexpected error.

getUnderlyingFileLinkInputStream

public static org.apache.hadoop.fs.FSDataInputStream getUnderlyingFileLinkInputStream(org.apache.hadoop.fs.FSDataInputStream stream)

If the passed FSDataInputStream is backed by a FileLink, returns the underlying InputStream for the resolved link target. Otherwise, returns null.

setLocations

protected void setLocations(org.apache.hadoop.fs.Path originPath,
                            org.apache.hadoop.fs.Path... alternativePaths)

NOTE: This method must be used only in the constructor! It creates a List with the specified locations for the link.

getBackReferencesDir

public static org.apache.hadoop.fs.Path getBackReferencesDir(org.apache.hadoop.fs.Path storeDir,
                                                             String fileName)

Get the directory to store the link back references

To simplify the reference count process, during the FileLink creation a back-reference is added to the back-reference directory of the specified file.

Parameters:

storeDir - Root directory for the link reference folder

fileName - File Name with links

Returns:

Path for the link back references.

getBackReferenceFileName

public static String getBackReferenceFileName(org.apache.hadoop.fs.Path dirPath)

Get the referenced file name from the reference link directory path.

Parameters:

dirPath - Link references directory path

Returns:

Name of the file referenced

isBackReferencesDir

public static boolean isBackReferencesDir(org.apache.hadoop.fs.Path dirPath)

Checks if the specified directory path is a back reference links folder.

Parameters:

dirPath - Directory path to verify

Returns:

True if the specified directory is a link references folder

equals

public boolean equals(Object obj)

Overrides:

equals in class Object

hashCode

public int hashCode()

Overrides:

hashCode in class Object