Package org.apache.hadoop.hbase.regionserver.storefiletracker

Class StoreFileListFile

java.lang.Object
org.apache.hadoop.hbase.regionserver.storefiletracker.StoreFileListFile
@Private class StoreFileListFile extends Object
To fully avoid listing, here we use two files for tracking. When loading, we will try to read both the two files, if only one exists, we will trust this one, if both exist, we will compare the timestamp to see which one is newer and trust that one. And we will record in memory that which one is trusted by us, and when we need to update the store file list, we will write to the other file.

So in this way, we could avoid listing when we want to load the store file list file.

To prevent loading partial file, we use the first 4 bytes as file length, and also append a 4 bytes crc32 checksum at the end. This is because the protobuf message parser sometimes can return without error on partial bytes if you stop at some special points, but the return message will have incorrect field value. We should try our best to prevent this happens because loading an incorrect store file list file usually leads to data loss.

To prevent failing silently while downgrading, where we may miss some newly introduced fields in StoreFileTrackerProtos.StoreFileList which are necessary, we introduce a 'version' field in StoreFileTrackerProtos.StoreFileList. If we find out that we are reading a StoreFileTrackerProtos.StoreFileList with higher version, we will fail immediately and tell users that you need extra steps while downgrading, to prevent potential data loss.

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    private final StoreContext
    ctx
     
    private static final org.slf4j.Logger
    LOG
     
    private static final int
    MAX_FILE_SIZE
     
    private int
    nextTrackFile
     
    private long
    prevTimestamp
     
    (package private) static final String
    TRACK_FILE_DIR
     
    (package private) static final Pattern
    TRACK_FILE_PATTERN
     
    (package private) static final String
    TRACK_FILE_PREFIX
     
    private static final String
    TRACK_FILE_ROTATE_PREFIX
     
    (package private) static final char
    TRACK_FILE_SEPARATOR
     
    private final org.apache.hadoop.fs.Path
    trackFileDir
     
    private final org.apache.hadoop.fs.Path[]
    trackFiles
     
    (package private) static final long
    VERSION
     

  • Constructor Summary

    Constructors
    Constructor
    Description
    StoreFileListFile(StoreContext ctx)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    private void
    cleanUpTrackFiles(long loadedSeqId, NavigableMap<Long,List<org.apache.hadoop.fs.Path>> seqId2TrackFiles)
     
    private void
    initializeTrackFiles(long seqId)
     
    private NavigableMap<Long,List<org.apache.hadoop.fs.Path>>
    listFiles()
     
    (package private) org.apache.hadoop.hbase.shaded.protobuf.generated.StoreFileTrackerProtos.StoreFileList
    load(boolean readOnly)
     
    (package private) static org.apache.hadoop.hbase.shaded.protobuf.generated.StoreFileTrackerProtos.StoreFileList
    load(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path)
     
    (package private) org.apache.hadoop.hbase.shaded.protobuf.generated.StoreFileTrackerProtos.StoreFileList
    load(org.apache.hadoop.fs.Path path)
     
    private int
    select(org.apache.hadoop.hbase.shaded.protobuf.generated.StoreFileTrackerProtos.StoreFileList[] lists)
     
    (package private) void
    update(org.apache.hadoop.hbase.shaded.protobuf.generated.StoreFileTrackerProtos.StoreFileList.Builder builder)
    We will set the timestamp and version in this method so just pass the builder in
    (package private) static void
    write(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path file, org.apache.hadoop.hbase.shaded.protobuf.generated.StoreFileTrackerProtos.StoreFileList storeFileList)
     

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait