Package org.apache.hadoop.hbase.mapreduce

Class WALInputFormat

java.lang.Object

org.apache.hadoop.mapreduce.InputFormat<WALKey,WALEdit>

org.apache.hadoop.hbase.mapreduce.WALInputFormat

@Public
public class WALInputFormat
extends org.apache.hadoop.mapreduce.InputFormat<WALKey,WALEdit>

Simple InputFormat for WAL files.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
(package private) static class	WALInputFormat.WALKeyRecordReader	handler for non-deprecated WALKey version.
(package private) static class	WALInputFormat.WALRecordReader<K extends WALKey>	RecordReader for an WAL file.
(package private) static class	WALInputFormat.WALSplit	InputSplit for WAL files.

Field Summary

Fields

Modifier and Type	Field	Description
static final String	END_TIME_KEY
private static final org.slf4j.Logger	LOG
static final String	START_TIME_KEY

Constructor Summary

Constructors

Constructor	Description
WALInputFormat()

Method Summary

Modifier and Type	Method	Description
(package private) static void	addFile(List<org.apache.hadoop.fs.FileStatus> result, org.apache.hadoop.fs.LocatedFileStatus lfs, long startTime, long endTime)
org.apache.hadoop.mapreduce.RecordReader<WALKey,WALEdit>	createRecordReader(org.apache.hadoop.mapreduce.InputSplit split, org.apache.hadoop.mapreduce.TaskAttemptContext context)
private List<org.apache.hadoop.fs.FileStatus>	getFiles(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path dir, long startTime, long endTime, org.apache.hadoop.conf.Configuration conf)
private org.apache.hadoop.fs.Path[]	getInputPaths(org.apache.hadoop.conf.Configuration conf)
List<org.apache.hadoop.mapreduce.InputSplit>	getSplits(org.apache.hadoop.mapreduce.JobContext context)
(package private) List<org.apache.hadoop.mapreduce.InputSplit>	getSplits(org.apache.hadoop.mapreduce.JobContext context, String startKey, String endKey)	implementation shared with deprecated HLogInputFormat
private static org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus>	listLocatedFileStatus(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path dir, org.apache.hadoop.conf.Configuration conf)	Attempts to return the LocatedFileStatus for the given directory.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

LOG

private static final org.slf4j.Logger LOG

START_TIME_KEY

public static final String START_TIME_KEY

See Also:

• Constant Field Values

END_TIME_KEY

public static final String END_TIME_KEY

See Also:

• Constant Field Values

Constructor Details

WALInputFormat

public WALInputFormat()

Method Details

getSplits

public List<org.apache.hadoop.mapreduce.InputSplit> getSplits(org.apache.hadoop.mapreduce.JobContext context)
                                                       throws IOException,
InterruptedException

Specified by:

getSplits in class org.apache.hadoop.mapreduce.InputFormat<WALKey,WALEdit>

Throws:

IOException

InterruptedException

getSplits

List<org.apache.hadoop.mapreduce.InputSplit> getSplits(org.apache.hadoop.mapreduce.JobContext context,
 String startKey,
 String endKey)
                                                throws IOException,
InterruptedException

implementation shared with deprecated HLogInputFormat

Throws:

IOException

InterruptedException

getInputPaths

private org.apache.hadoop.fs.Path[] getInputPaths(org.apache.hadoop.conf.Configuration conf)

getFiles

private List<org.apache.hadoop.fs.FileStatus> getFiles(org.apache.hadoop.fs.FileSystem fs,
 org.apache.hadoop.fs.Path dir,
 long startTime,
 long endTime,
 org.apache.hadoop.conf.Configuration conf)
                                                throws IOException

Parameters:

startTime - If file looks like it has a timestamp in its name, we'll check if newer or equal to this value else we will filter out the file. If name does not seem to have a timestamp, we will just return it w/o filtering.

endTime - If file looks like it has a timestamp in its name, we'll check if older or equal to this value else we will filter out the file. If name does not seem to have a timestamp, we will just return it w/o filtering.

Throws:

IOException

addFile

static void addFile(List<org.apache.hadoop.fs.FileStatus> result,
 org.apache.hadoop.fs.LocatedFileStatus lfs,
 long startTime,
 long endTime)

createRecordReader

public org.apache.hadoop.mapreduce.RecordReader<WALKey,WALEdit> createRecordReader(org.apache.hadoop.mapreduce.InputSplit split,
 org.apache.hadoop.mapreduce.TaskAttemptContext context)
                                                                            throws IOException,
InterruptedException

Specified by:

createRecordReader in class org.apache.hadoop.mapreduce.InputFormat<WALKey,WALEdit>

Throws:

IOException

InterruptedException

listLocatedFileStatus

private static org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> listLocatedFileStatus(org.apache.hadoop.fs.FileSystem fs,
 org.apache.hadoop.fs.Path dir,
 org.apache.hadoop.conf.Configuration conf)
                                                                                                          throws IOException

Attempts to return the LocatedFileStatus for the given directory. If the directory does not exist, it will check if the directory is an archived log file and try to find it

Throws:

IOException