Package org.apache.hadoop.hbase.mapreduce

Class WALInputFormat

java.lang.Object

org.apache.hadoop.mapreduce.InputFormat<WALKey,WALEdit>

org.apache.hadoop.hbase.mapreduce.WALInputFormat

@Public
public class WALInputFormat
extends org.apache.hadoop.mapreduce.InputFormat<WALKey,WALEdit>

Simple InputFormat for WAL files.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
(package private) static class	WALInputFormat.WALKeyRecordReader	handler for non-deprecated WALKey version.
(package private) static class	WALInputFormat.WALRecordReader<K extends WALKey>	RecordReader for an WAL file.
static class	WALInputFormat.WALSplit	InputSplit for WAL files.

Field Summary

Fields

Modifier and Type	Field	Description
static final String	END_TIME_KEY
private static final org.slf4j.Logger	LOG
static final String	START_TIME_KEY

Constructor Summary

Constructors

Constructor	Description
WALInputFormat()

Method Summary

Modifier and Type	Method	Description
(package private) static void	addFile(List<org.apache.hadoop.fs.FileStatus> result, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.LocatedFileStatus lfs, long startTime, long endTime)
org.apache.hadoop.mapreduce.RecordReader<WALKey,WALEdit>	createRecordReader(org.apache.hadoop.mapreduce.InputSplit split, org.apache.hadoop.mapreduce.TaskAttemptContext context)
(package private) List<org.apache.hadoop.fs.FileStatus>	getFiles(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path dir, long startTime, long endTime, org.apache.hadoop.conf.Configuration conf)
(package private) org.apache.hadoop.fs.Path[]	getInputPaths(org.apache.hadoop.conf.Configuration conf)
List<org.apache.hadoop.mapreduce.InputSplit>	getSplits(org.apache.hadoop.mapreduce.JobContext context)
(package private) List<org.apache.hadoop.mapreduce.InputSplit>	getSplits(org.apache.hadoop.mapreduce.JobContext context, String startKey, String endKey)	implementation shared with deprecated HLogInputFormat
private static boolean	isClosedBefore(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.LocatedFileStatus lfs, long time)	Whether the file is closed and its final modification time precedes time.
private static org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus>	listLocatedFileStatus(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path dir, org.apache.hadoop.conf.Configuration conf)	Attempts to return the LocatedFileStatus for the given directory.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

LOG

private static final org.slf4j.Logger LOG

START_TIME_KEY

public static final String START_TIME_KEY

See Also:

• Constant Field Values

END_TIME_KEY

public static final String END_TIME_KEY

See Also:

• Constant Field Values

Constructor Details

WALInputFormat

public WALInputFormat()

Method Details

getSplits

public List<org.apache.hadoop.mapreduce.InputSplit> getSplits(org.apache.hadoop.mapreduce.JobContext context)
                                                       throws IOException,
InterruptedException

Specified by:

getSplits in class org.apache.hadoop.mapreduce.InputFormat<WALKey,WALEdit>

Throws:

IOException

InterruptedException

getSplits

List<org.apache.hadoop.mapreduce.InputSplit> getSplits(org.apache.hadoop.mapreduce.JobContext context,
 String startKey,
 String endKey)
                                                throws IOException,
InterruptedException

implementation shared with deprecated HLogInputFormat

Throws:

IOException

InterruptedException

getInputPaths

org.apache.hadoop.fs.Path[] getInputPaths(org.apache.hadoop.conf.Configuration conf)

getFiles

List<org.apache.hadoop.fs.FileStatus> getFiles(org.apache.hadoop.fs.FileSystem fs,
 org.apache.hadoop.fs.Path dir,
 long startTime,
 long endTime,
 org.apache.hadoop.conf.Configuration conf)
                                        throws IOException

Parameters:

startTime - Files created before this time are dropped only if confirmed closed before it. Files without a parseable timestamp in their name are always included.

endTime - Files created after this time are dropped. Files without a parseable timestamp in their name are always included.

Throws:

IOException

isClosedBefore

private static boolean isClosedBefore(org.apache.hadoop.fs.FileSystem fs,
 org.apache.hadoop.fs.LocatedFileStatus lfs,
 long time)

Whether the file is closed and its final modification time precedes time. Only a closed file has a reliable modification time, so an open file or a non-HDFS file always returns false (kept). When the file is confirmed closed, its status is re-fetched because the lfs from listLocatedStatus may carry a stale creation-time mtime from when the file was still open.

addFile

static void addFile(List<org.apache.hadoop.fs.FileStatus> result,
 org.apache.hadoop.fs.FileSystem fs,
 org.apache.hadoop.fs.LocatedFileStatus lfs,
 long startTime,
 long endTime)

createRecordReader

public org.apache.hadoop.mapreduce.RecordReader<WALKey,WALEdit> createRecordReader(org.apache.hadoop.mapreduce.InputSplit split,
 org.apache.hadoop.mapreduce.TaskAttemptContext context)
                                                                            throws IOException,
InterruptedException

Specified by:

createRecordReader in class org.apache.hadoop.mapreduce.InputFormat<WALKey,WALEdit>

Throws:

IOException

InterruptedException

listLocatedFileStatus

private static org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> listLocatedFileStatus(org.apache.hadoop.fs.FileSystem fs,
 org.apache.hadoop.fs.Path dir,
 org.apache.hadoop.conf.Configuration conf)
                                                                                                          throws IOException

Attempts to return the LocatedFileStatus for the given directory. If the directory does not exist, it will check if the directory is an archived log file and try to find it

Throws:

IOException