org.apache.hadoop.hbase.regionserver.wal

Class ProtobufLogReader

java.lang.Object

org.apache.hadoop.hbase.regionserver.wal.ReaderBase

org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader

All Implemented Interfaces:

Closeable, AutoCloseable, AbstractFSWALProvider.Reader, WAL.Reader

Direct Known Subclasses:

SecureProtobufLogReader

@InterfaceAudience.LimitedPrivate(value={"Coprocesssor","Phoenix","Configuration"})
public class ProtobufLogReader
extends ReaderBase

A Protobuf based WAL has the following structure:

<PB_WAL_MAGIC><WALHeader><WALEdits>...<WALEdits><Trailer> <TrailerSize> <PB_WAL_COMPLETE_MAGIC>

The Reader reads meta information (WAL Compression state, WALTrailer, etc) in ProtobufLogReader#initReader(FSDataInputStream). A WALTrailer is an extensible structure which is appended at the end of the WAL. This is empty for now; it can contain some meta information such as Region level stats, etc in future.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
(package private) static class	ProtobufLogReader.WALHdrContext
(package private) static class	ProtobufLogReader.WALHdrResult

Field Summary

Fields

Modifier and Type	Field and Description
protected WALCellCodec.ByteStringUncompressor	byteStringUncompressor
protected Codec.Decoder	cellDecoder
private String	codecClsName
(package private) static int	DEFAULT_WAL_TRAILER_WARN_SIZE
protected boolean	hasCompression
protected boolean	hasTagCompression
protected boolean	hasValueCompression
protected org.apache.hadoop.fs.FSDataInputStream	inputStream
private static org.slf4j.Logger	LOG
static byte[]	PB_WAL_COMPLETE_MAGIC
static byte[]	PB_WAL_MAGIC
protected org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos.WALTrailer	trailer
private boolean	trailerPresent
protected int	trailerWarnSize
protected Compression.Algorithm	valueCompressionType
(package private) static String	WAL_TRAILER_WARN_SIZE Configuration name of WAL Trailer's warning size.
private long	walEditsStopOffset
private static List<String>	writerClsNames

Fields inherited from class org.apache.hadoop.hbase.regionserver.wal.ReaderBase

compressionContext, conf, edit, emptyCompressionContext, fileLength, fs, path

Constructor Summary

Constructors

Constructor and Description
ProtobufLogReader()

Method Summary

Modifier and Type	Method and Description
void	close()
private IOException	extractHiddenEof(Exception ex)
protected WALCellCodec	getCodec(org.apache.hadoop.conf.Configuration conf, String cellCodecClsName, CompressionContext compressionContext)
String	getCodecClsName()
long	getPosition()
protected Compression.Algorithm	getValueCompressionAlgorithm() Returns Value compression algorithm for this log.
List<String>	getWriterClsNames()
protected boolean	hasCompression() Returns Whether compression is enabled for this log.
protected boolean	hasTagCompression() Returns Whether tag compression is enabled for this log.
protected boolean	hasValueCompression() Returns Whether value compression is enabled for this log.
void	init(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FSDataInputStream stream)
protected void	initAfterCompression() Initializes the compression after the shared stuff has been initialized.
protected void	initAfterCompression(String cellCodecClsName) Initializes the compression after the shared stuff has been initialized.
private String	initInternal(org.apache.hadoop.fs.FSDataInputStream stream, boolean isFirst)
protected String	initReader(org.apache.hadoop.fs.FSDataInputStream stream) Initializes the log reader with a particular stream (may be null).
protected ProtobufLogReader.WALHdrContext	readHeader(org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos.WALHeader.Builder builder, org.apache.hadoop.fs.FSDataInputStream stream)
protected boolean	readNext(WAL.Entry entry) Read next entry.
void	reset()
protected void	seekOnFs(long pos) Performs a filesystem-level seek to a certain position in an underlying file.
private boolean	setTrailerIfPresent() To check whether a trailer is present in a WAL, it seeks to position (fileLength - PB_WAL_COMPLETE_MAGIC.size() - Bytes.SIZEOF_INT).
long	trailerSize()

Methods inherited from class org.apache.hadoop.hbase.regionserver.wal.ReaderBase

next, next, seek

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

private static final org.slf4j.Logger LOG

PB_WAL_MAGIC

@InterfaceAudience.Private
public static final byte[] PB_WAL_MAGIC

PB_WAL_COMPLETE_MAGIC

@InterfaceAudience.Private
public static final byte[] PB_WAL_COMPLETE_MAGIC

WAL_TRAILER_WARN_SIZE

static final String WAL_TRAILER_WARN_SIZE

Configuration name of WAL Trailer's warning size. If a waltrailer's size is greater than the configured size, providers should log a warning. e.g. this is used with Protobuf reader/writer.

See Also:

Constant Field Values

DEFAULT_WAL_TRAILER_WARN_SIZE

static final int DEFAULT_WAL_TRAILER_WARN_SIZE

See Also:

Constant Field Values

inputStream

protected org.apache.hadoop.fs.FSDataInputStream inputStream

cellDecoder

protected Codec.Decoder cellDecoder

byteStringUncompressor

protected WALCellCodec.ByteStringUncompressor byteStringUncompressor

hasCompression

protected boolean hasCompression

hasTagCompression

protected boolean hasTagCompression

hasValueCompression

protected boolean hasValueCompression

valueCompressionType

protected Compression.Algorithm valueCompressionType

walEditsStopOffset

private long walEditsStopOffset

trailerPresent

private boolean trailerPresent

trailer

protected org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos.WALTrailer trailer

trailerWarnSize

protected int trailerWarnSize

writerClsNames

private static List<String> writerClsNames

codecClsName

private String codecClsName

Constructor Detail

ProtobufLogReader

public ProtobufLogReader()

Method Detail

trailerSize

@InterfaceAudience.Private
public long trailerSize()

close

public void close()
           throws IOException

Throws:

IOException

getPosition

public long getPosition()
                 throws IOException

Throws:

IOException

reset

public void reset()
           throws IOException

Throws:

IOException

init

public void init(org.apache.hadoop.fs.FileSystem fs,
                 org.apache.hadoop.fs.Path path,
                 org.apache.hadoop.conf.Configuration conf,
                 org.apache.hadoop.fs.FSDataInputStream stream)
          throws IOException

Specified by:

init in interface AbstractFSWALProvider.Reader

Overrides:

init in class ReaderBase

Parameters:

fs - File system.

path - Path.

conf - Configuration.

stream - Input stream that may have been pre-opened by the caller; may be null.

Throws:

IOException

initReader

protected String initReader(org.apache.hadoop.fs.FSDataInputStream stream)
                     throws IOException

Description copied from class: ReaderBase

Initializes the log reader with a particular stream (may be null). Reader assumes ownership of the stream if not null and may use it. Called once.

Specified by:

initReader in class ReaderBase

Returns:

the class name of cell Codec, null if such information is not available

Throws:

IOException

getWriterClsNames

public List<String> getWriterClsNames()

getCodecClsName

public String getCodecClsName()

readHeader

protected ProtobufLogReader.WALHdrContext readHeader(org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos.WALHeader.Builder builder,
                                                     org.apache.hadoop.fs.FSDataInputStream stream)
                                              throws IOException

Throws:

IOException

initInternal

private String initInternal(org.apache.hadoop.fs.FSDataInputStream stream,
                            boolean isFirst)
                     throws IOException

Throws:

IOException

setTrailerIfPresent

private boolean setTrailerIfPresent()

To check whether a trailer is present in a WAL, it seeks to position (fileLength - PB_WAL_COMPLETE_MAGIC.size() - Bytes.SIZEOF_INT). It reads the int value to know the size of the trailer, and checks whether the trailer is present at the end or not by comparing the last PB_WAL_COMPLETE_MAGIC.size() bytes. In case trailer is not present, it returns false; otherwise, sets the trailer and sets this.walEditsStopOffset variable up to the point just before the trailer.

The trailer is ignored in case:
• fileLength is 0 or not correct (when file is under recovery, etc).
• the trailer size is negative.

In case the trailer size > this.trailerMaxSize, it is read after a WARN message.

Returns:

true if a valid trailer is present n

getCodec

protected WALCellCodec getCodec(org.apache.hadoop.conf.Configuration conf,
                                String cellCodecClsName,
                                CompressionContext compressionContext)
                         throws IOException

Throws:

IOException

initAfterCompression

protected void initAfterCompression()
                             throws IOException

Description copied from class: ReaderBase

Initializes the compression after the shared stuff has been initialized. Called once.

Specified by:

initAfterCompression in class ReaderBase

Throws:

IOException

initAfterCompression

protected void initAfterCompression(String cellCodecClsName)
                             throws IOException

Description copied from class: ReaderBase

Initializes the compression after the shared stuff has been initialized. Called once.

Specified by:

initAfterCompression in class ReaderBase

Parameters:

cellCodecClsName - class name of cell Codec

Throws:

IOException

hasCompression

protected boolean hasCompression()

Description copied from class: ReaderBase

Returns Whether compression is enabled for this log.

Specified by:

hasCompression in class ReaderBase

hasTagCompression

protected boolean hasTagCompression()

Description copied from class: ReaderBase

Returns Whether tag compression is enabled for this log.

Specified by:

hasTagCompression in class ReaderBase

hasValueCompression

protected boolean hasValueCompression()

Description copied from class: ReaderBase

Returns Whether value compression is enabled for this log.

Specified by:

hasValueCompression in class ReaderBase

getValueCompressionAlgorithm

protected Compression.Algorithm getValueCompressionAlgorithm()

Description copied from class: ReaderBase

Returns Value compression algorithm for this log.

Specified by:

getValueCompressionAlgorithm in class ReaderBase

readNext

protected boolean readNext(WAL.Entry entry)
                    throws IOException

Description copied from class: ReaderBase

Read next entry.

Specified by:

readNext in class ReaderBase

Parameters:

entry - The entry to read into.

Returns:

Whether there was anything to read.

Throws:

IOException

extractHiddenEof

private IOException extractHiddenEof(Exception ex)

seekOnFs

protected void seekOnFs(long pos)
                 throws IOException

Description copied from class: ReaderBase

Performs a filesystem-level seek to a certain position in an underlying file.

Specified by:

seekOnFs in class ReaderBase

Throws:

IOException