org.apache.hadoop.hbase.regionserver.wal

Class AsyncProtobufLogWriter

java.lang.Object

org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter

org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter

All Implemented Interfaces:

Closeable, AutoCloseable, AsyncFSWALProvider.AsyncWriter, WALProvider.AsyncWriter, WALProvider.WriterBase

Direct Known Subclasses:

SecureAsyncProtobufLogWriter

@InterfaceAudience.Private
public class AsyncProtobufLogWriter
extends AbstractProtobufLogWriter
implements AsyncFSWALProvider.AsyncWriter

AsyncWriter for protobuf-based WAL.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
private static class	AsyncProtobufLogWriter.OutputStreamWrapper

Field Summary

Fields

Modifier and Type	Field and Description
private OutputStream	asyncOutputWrapper
private Class<? extends org.apache.hbase.thirdparty.io.netty.channel.Channel>	channelClass
private org.apache.hbase.thirdparty.io.netty.channel.EventLoopGroup	eventLoopGroup
private long	finalSyncedLength Save AsyncFSOutput.getSyncedLength() when output is closed.
private static org.slf4j.Logger	LOG
private AsyncFSOutput	output
private long	waitTimeout

Fields inherited from class org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter

cellEncoder, compressionContext, compressor, conf, length, trailer, trailerWarnSize, trailerWritten

Constructor Summary

Constructors

Constructor and Description
AsyncProtobufLogWriter(org.apache.hbase.thirdparty.io.netty.channel.EventLoopGroup eventLoopGroup, Class<? extends org.apache.hbase.thirdparty.io.netty.channel.Channel> channelClass)

Method Summary

Modifier and Type	Method and Description
void	append(WAL.Entry entry)
void	close()
protected void	closeOutputIfNecessary() It is straight forward to close the output, do not need to write trailer like the Writer.close
AsyncFSOutput	getOutput()
protected OutputStream	getOutputStreamForCellEncoder()
long	getSyncedLength() NOTE: We add this method for WALFileLengthProvider used for replication, considering the case if we use AsyncFSWAL,we write to 3 DNs concurrently, according to the visibility guarantee of HDFS, the data will be available immediately when arriving at DN since all the DNs will be considered as the last one in pipeline.
protected String	getWriterClassName()
protected void	initOutput(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, boolean overwritable, int bufferSize, short replication, long blockSize, StreamSlowMonitor monitor)
CompletableFuture<Long>	sync(boolean forceSync)
protected long	writeMagicAndWALHeader(byte[] magic, org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos.WALHeader header) return the file length after written.
private long	writeWALMetadata(Consumer<CompletableFuture<Long>> action)
protected long	writeWALTrailerAndMagic(org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos.WALTrailer trailer, byte[] magic)

Methods inherited from class org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter

buildSecureWALHeader, buildWALHeader, getLength, init, initAfterHeader, secureInitAfterHeader, setEncryptor, setWALTrailer, writeWALTrailer

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.hbase.wal.AsyncFSWALProvider.AsyncWriter

init

Methods inherited from interface org.apache.hadoop.hbase.wal.WALProvider.WriterBase

getLength

Field Detail

LOG

private static final org.slf4j.Logger LOG

eventLoopGroup

private final org.apache.hbase.thirdparty.io.netty.channel.EventLoopGroup eventLoopGroup

channelClass

private final Class<? extends org.apache.hbase.thirdparty.io.netty.channel.Channel> channelClass

output

private volatile AsyncFSOutput output

finalSyncedLength

private volatile long finalSyncedLength

Save AsyncFSOutput.getSyncedLength() when output is closed.

asyncOutputWrapper

private OutputStream asyncOutputWrapper

waitTimeout

private long waitTimeout

Constructor Detail

AsyncProtobufLogWriter

public AsyncProtobufLogWriter(org.apache.hbase.thirdparty.io.netty.channel.EventLoopGroup eventLoopGroup,
                              Class<? extends org.apache.hbase.thirdparty.io.netty.channel.Channel> channelClass)

Method Detail

getWriterClassName

protected String getWriterClassName()

Overrides:

getWriterClassName in class AbstractProtobufLogWriter

append

public void append(WAL.Entry entry)

Specified by:

append in interface WALProvider.AsyncWriter

sync

public CompletableFuture<Long> sync(boolean forceSync)

Specified by:

sync in interface WALProvider.AsyncWriter

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException

getOutput

public AsyncFSOutput getOutput()

initOutput

protected void initOutput(org.apache.hadoop.fs.FileSystem fs,
                          org.apache.hadoop.fs.Path path,
                          boolean overwritable,
                          int bufferSize,
                          short replication,
                          long blockSize,
                          StreamSlowMonitor monitor)
                   throws IOException,
                          CommonFSUtils.StreamLacksCapabilityException

Specified by:

initOutput in class AbstractProtobufLogWriter

Throws:

IOException

CommonFSUtils.StreamLacksCapabilityException

closeOutputIfNecessary

protected void closeOutputIfNecessary()

Description copied from class: AbstractProtobufLogWriter

It is straight forward to close the output, do not need to write trailer like the Writer.close

Overrides:

closeOutputIfNecessary in class AbstractProtobufLogWriter

writeWALMetadata

private long writeWALMetadata(Consumer<CompletableFuture<Long>> action)
                       throws IOException

Throws:

IOException

writeMagicAndWALHeader

protected long writeMagicAndWALHeader(byte[] magic,
                                      org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos.WALHeader header)
                               throws IOException

Description copied from class: AbstractProtobufLogWriter

return the file length after written.

Specified by:

writeMagicAndWALHeader in class AbstractProtobufLogWriter

Throws:

IOException

writeWALTrailerAndMagic

protected long writeWALTrailerAndMagic(org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos.WALTrailer trailer,
                                       byte[] magic)
                                throws IOException

Specified by:

writeWALTrailerAndMagic in class AbstractProtobufLogWriter

Throws:

IOException

getOutputStreamForCellEncoder

protected OutputStream getOutputStreamForCellEncoder()

Specified by:

getOutputStreamForCellEncoder in class AbstractProtobufLogWriter

getSyncedLength

public long getSyncedLength()

Description copied from interface: WALProvider.WriterBase

NOTE: We add this method for WALFileLengthProvider used for replication, considering the case if we use AsyncFSWAL,we write to 3 DNs concurrently, according to the visibility guarantee of HDFS, the data will be available immediately when arriving at DN since all the DNs will be considered as the last one in pipeline. This means replication may read uncommitted data and replicate it to the remote cluster and cause data inconsistency. The method WALProvider.WriterBase.getLength() may return length which just in hdfs client buffer and not successfully synced to HDFS, so we use this method to return the length successfully synced to HDFS and replication thread could only read writing WAL file limited by this length. see also HBASE-14004 and this document for more details: https://docs.google.com/document/d/11AyWtGhItQs6vsLRIx32PwTxmBY3libXwGXI25obVEY/edit#

Specified by:

getSyncedLength in interface WALProvider.WriterBase

Returns:

byteSize successfully synced to underlying filesystem.