org.apache.hadoop.hbase.regionserver.wal

Class ProtobufLogWriter

java.lang.Object

org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter

org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter

All Implemented Interfaces:

Closeable, AutoCloseable, FSHLogProvider.Writer, WALProvider.Writer, WALProvider.WriterBase

Direct Known Subclasses:

SecureProtobufLogWriter

@InterfaceAudience.Private
public class ProtobufLogWriter
extends AbstractProtobufLogWriter
implements FSHLogProvider.Writer

Writer for protobuf-based WAL.

Field Summary

Fields

Modifier and Type	Field and Description
private static org.slf4j.Logger	LOG
protected org.apache.hadoop.fs.FSDataOutputStream	output
private AtomicLong	syncedLength

Fields inherited from class org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter

cellEncoder, compressionContext, compressor, conf, length, trailer, trailerWarnSize, trailerWritten

Constructor Summary

Constructors

Constructor and Description
ProtobufLogWriter()

Method Summary

Modifier and Type	Method and Description
void	append(WAL.Entry entry)
void	close()
protected void	closeOutputIfNecessary() It is straight forward to close the output, do not need to write trailer like the Writer.close
protected OutputStream	getOutputStreamForCellEncoder()
org.apache.hadoop.fs.FSDataOutputStream	getStream()
long	getSyncedLength() NOTE: We add this method for WALFileLengthProvider used for replication, considering the case if we use AsyncFSWAL,we write to 3 DNs concurrently, according to the visibility guarantee of HDFS, the data will be available immediately when arriving at DN since all the DNs will be considered as the last one in pipeline.
protected void	initOutput(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, boolean overwritable, int bufferSize, short replication, long blockSize, StreamSlowMonitor monitor)
void	sync(boolean forceSync)
protected long	writeMagicAndWALHeader(byte[] magic, org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos.WALHeader header) return the file length after written.
protected long	writeWALTrailerAndMagic(org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos.WALTrailer trailer, byte[] magic)

Methods inherited from class org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter

buildSecureWALHeader, buildWALHeader, getLength, getWriterClassName, init, initAfterHeader, secureInitAfterHeader, setEncryptor, setWALTrailer, writeWALTrailer

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.hbase.wal.FSHLogProvider.Writer

init

Methods inherited from interface org.apache.hadoop.hbase.wal.WALProvider.WriterBase

getLength

Field Detail

LOG

private static final org.slf4j.Logger LOG

output

protected org.apache.hadoop.fs.FSDataOutputStream output

syncedLength

private final AtomicLong syncedLength

Constructor Detail

ProtobufLogWriter

public ProtobufLogWriter()

Method Detail

append

public void append(WAL.Entry entry)
            throws IOException

Specified by:

append in interface WALProvider.Writer

Throws:

IOException

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException

sync

public void sync(boolean forceSync)
          throws IOException

Specified by:

sync in interface WALProvider.Writer

Throws:

IOException

getSyncedLength

public long getSyncedLength()

Description copied from interface: WALProvider.WriterBase

NOTE: We add this method for WALFileLengthProvider used for replication, considering the case if we use AsyncFSWAL,we write to 3 DNs concurrently, according to the visibility guarantee of HDFS, the data will be available immediately when arriving at DN since all the DNs will be considered as the last one in pipeline. This means replication may read uncommitted data and replicate it to the remote cluster and cause data inconsistency. The method WALProvider.WriterBase.getLength() may return length which just in hdfs client buffer and not successfully synced to HDFS, so we use this method to return the length successfully synced to HDFS and replication thread could only read writing WAL file limited by this length. see also HBASE-14004 and this document for more details: https://docs.google.com/document/d/11AyWtGhItQs6vsLRIx32PwTxmBY3libXwGXI25obVEY/edit#

Specified by:

getSyncedLength in interface WALProvider.WriterBase

Returns:

byteSize successfully synced to underlying filesystem.

getStream

public org.apache.hadoop.fs.FSDataOutputStream getStream()

initOutput

protected void initOutput(org.apache.hadoop.fs.FileSystem fs,
                          org.apache.hadoop.fs.Path path,
                          boolean overwritable,
                          int bufferSize,
                          short replication,
                          long blockSize,
                          StreamSlowMonitor monitor)
                   throws IOException,
                          CommonFSUtils.StreamLacksCapabilityException

Specified by:

initOutput in class AbstractProtobufLogWriter

Throws:

IOException

CommonFSUtils.StreamLacksCapabilityException

closeOutputIfNecessary

protected void closeOutputIfNecessary()

Description copied from class: AbstractProtobufLogWriter

It is straight forward to close the output, do not need to write trailer like the Writer.close

Overrides:

closeOutputIfNecessary in class AbstractProtobufLogWriter

writeMagicAndWALHeader

protected long writeMagicAndWALHeader(byte[] magic,
                                      org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos.WALHeader header)
                               throws IOException

Description copied from class: AbstractProtobufLogWriter

return the file length after written.

Specified by:

writeMagicAndWALHeader in class AbstractProtobufLogWriter

Throws:

IOException

getOutputStreamForCellEncoder

protected OutputStream getOutputStreamForCellEncoder()

Specified by:

getOutputStreamForCellEncoder in class AbstractProtobufLogWriter

writeWALTrailerAndMagic

protected long writeWALTrailerAndMagic(org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos.WALTrailer trailer,
                                       byte[] magic)
                                throws IOException

Specified by:

writeWALTrailerAndMagic in class AbstractProtobufLogWriter

Throws:

IOException