Package org.apache.hadoop.hbase.test.util.warc

Class WARCRecord

java.lang.Object

org.apache.hadoop.hbase.test.util.warc.WARCRecord

public class WARCRecord
extends Object

Immutable implementation of a record in a WARC file. You create a WARCRecord by parsing it out of a DataInput stream.

The file format is documented in the ISO Standard. In a nutshell, it's a textual format consisting of lines delimited by `\r\n`. Each record has the following structure:

1. A line indicating the WARC version number, such as `WARC/1.0`.
2. Several header lines (in key-value format, similar to HTTP or email headers), giving information about the record. The header is terminated by an empty line.
3. A body consisting of raw bytes (the number of bytes is indicated in one of the headers).
4. A final separator of `\r\n\r\n` before the next record starts.

There are various different types of records, as documented on WARCRecord.Header.getRecordType().

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static final class	WARCRecord.Header	Contains the parsed headers of a WARCRecord.

Field Summary

Fields

Modifier and Type	Field	Description
private final byte[]	content
private static final Pattern	CONTINUATION_PATTERN
private static final String	CRLF
private static final byte[]	CRLF_BYTES
private final WARCRecord.Header	header
private static final Pattern	VERSION_PATTERN
static final String	WARC_VERSION

Constructor Summary

Constructors

Constructor	Description
WARCRecord(DataInput in)	Creates a new WARCRecord by parsing it out of a DataInput stream.

Method Summary

Modifier and Type	Method	Description
byte[]	getContent()	Returns the body of the record, as an unparsed raw array of bytes.
WARCRecord.Header	getHeader()	Returns the parsed header structure of the WARC record.
private static WARCRecord.Header	readHeader(DataInput in)
private static String	readLine(DataInput in)
private static void	readSeparator(DataInput in)
String	toString()	Returns a human-readable string representation of the record.
void	write(DataOutput out)	Writes this record to a DataOutput stream.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Details

WARC_VERSION

public static final String WARC_VERSION

See Also:

• Constant Field Values

VERSION_PATTERN

private static final Pattern VERSION_PATTERN

CONTINUATION_PATTERN

private static final Pattern CONTINUATION_PATTERN

CRLF

private static final String CRLF

See Also:

• Constant Field Values

CRLF_BYTES

private static final byte[] CRLF_BYTES

header

private final WARCRecord.Header header

content

private final byte[] content

Constructor Details

WARCRecord

public WARCRecord(DataInput in)
           throws IOException

Creates a new WARCRecord by parsing it out of a DataInput stream.

Parameters:

in - The input source from which one record will be read.

Throws:

IOException

Method Details

readHeader

private static WARCRecord.Header readHeader(DataInput in)
                                     throws IOException

Throws:

IOException

readLine

private static String readLine(DataInput in)
                        throws IOException

Throws:

IOException

readSeparator

private static void readSeparator(DataInput in)
                           throws IOException

Throws:

IOException

getHeader

public WARCRecord.Header getHeader()

Returns the parsed header structure of the WARC record.

getContent

public byte[] getContent()

Returns the body of the record, as an unparsed raw array of bytes. The content of the body depends on the type of record (see WARCRecord.Header.getRecordType()). For example, in the case of a `response` type header, the body consists of the full HTTP response returned by the server (HTTP headers followed by the body).

write

public void write(DataOutput out)
           throws IOException

Writes this record to a DataOutput stream. The output may, in some edge cases, be not byte-for-byte identical to what was parsed from a DataInput. However it has the same meaning and should not lose any information.

Parameters:

out - The output stream to which this record should be appended.

Throws:

IOException

toString

public String toString()

Returns a human-readable string representation of the record.

Overrides:

toString in class Object