Package org.apache.hadoop.hbase.regionserver.wal

Class AbstractProtobufWALReader

java.lang.Object
org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufWALReader
All Implemented Interfaces:
Closeable, AutoCloseable, AbstractFSWALProvider.Initializer
Direct Known Subclasses:
ProtobufWALStreamReader, ProtobufWALTailingReader
@Private public abstract class AbstractProtobufWALReader extends Object implements AbstractFSWALProvider.Initializer, Closeable
Base class for reading protobuf based wal reader

  • Field Details

  • Constructor Details

  • Method Details

    • getCellCodecInputStream

      protected abstract InputStream getCellCodecInputStream(org.apache.hadoop.fs.FSDataInputStream stream)
      Get or create the input stream used by cell decoder.

      For implementing replication, we may need to limit the bytes we can read, so here we provide a method so subclasses can wrap the original input stream.

    • skipTo

      protected abstract void skipTo(long position) throws IOException
      Skip to the given position.
      Throws:
      IOException

    • init

      public void init(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration conf, long startPosition) throws IOException
      Description copied from interface: AbstractFSWALProvider.Initializer
      A method to initialize a WAL reader.
      Specified by:
      init in interface AbstractFSWALProvider.Initializer
      startPosition - the start position you want to read from, -1 means start reading from the first WAL entry. Notice that, the first entry is not started at position as we have several headers, so typically you should not pass 0 here.
      Throws:
      IOException

    • openArchivedWAL

      private Pair<org.apache.hadoop.fs.FSDataInputStream,org.apache.hadoop.fs.FileStatus> openArchivedWAL() throws IOException
      Throws:
      IOException

    • open

      protected final Pair<org.apache.hadoop.fs.FSDataInputStream,org.apache.hadoop.fs.FileStatus> open() throws IOException
      Throws:
      IOException

    • readHeader

      protected final org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos.WALHeader readHeader(org.apache.hadoop.fs.FSDataInputStream stream) throws IOException
      Throws:
      IOException

    • initDecryptor

      private void initDecryptor(org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos.WALHeader header) throws IOException
      Throws:
      IOException

    • initCompression

      private void initCompression(org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos.WALHeader header) throws IOException
      Throws:
      IOException

    • getCodec

      private WALCellCodec getCodec(org.apache.hadoop.conf.Configuration conf, String cellCodecClsName, CompressionContext compressionContext) throws IOException
      Throws:
      IOException

    • initWALCellCodec

      protected final void initWALCellCodec(org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos.WALHeader header, InputStream inputStream) throws IOException
      Throws:
      IOException

    • readTrailer

      protected final void readTrailer(org.apache.hadoop.fs.FSDataInputStream stream, org.apache.hadoop.fs.FileStatus stat) throws IOException
      Throws:
      IOException

    • setTrailerIfPresent

      private boolean setTrailerIfPresent(org.apache.hadoop.fs.FSDataInputStream stream) throws IOException
      To check whether a trailer is present in a WAL, it seeks to position (fileLength - PB_WAL_COMPLETE_MAGIC.size() - Bytes.SIZEOF_INT). It reads the int value to know the size of the trailer, and checks whether the trailer is present at the end or not by comparing the last PB_WAL_COMPLETE_MAGIC.size() bytes. In case trailer is not present, it returns false; otherwise, sets the trailer and sets this.walEditsStopOffset variable up to the point just before the trailer.

      The trailer is ignored in case:

      • fileLength is 0 or not correct (when file is under recovery, etc).
      • the trailer size is negative.
      In case the trailer size > this.trailerMaxSize, it is read after a WARN message.
      Returns:
      true if a valid trailer is present
      Throws:
      IOException

    • reachWALEditsStopOffset

      protected final boolean reachWALEditsStopOffset(long pos)

    • getWriterClsNames

      public List<String> getWriterClsNames()
      Returns names of the accepted writer classes

    • getCodecClsName

      public String getCodecClsName()
      Returns the cell codec classname

    • getPosition

      public long getPosition() throws IOException
      Throws:
      IOException

    • trailerSize

      public long trailerSize()

    • getPositionQuietly

      protected final String getPositionQuietly()

    • extractHiddenEof

      protected final IOException extractHiddenEof(Exception ex)

    • isWALTrailer

      protected final boolean isWALTrailer(long startPosition) throws IOException
      This is used to determine whether we have already reached the WALTrailer. As the size and magic are at the end of the WAL file, it is possible that these two options are missing while writing, so we will consider there is no trailer. And when we actually reach the WALTrailer, we will try to decode it as WALKey and we will fail but the error could be varied as it is parsing WALTrailer actually.
      Returns:
      whether this is a WALTrailer and we should throw EOF to upper layer the file is done
      Throws:
      IOException

    • close

      public void close()
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable