org.apache.hadoop.hbase.regionserver.storefiletracker

Class StoreFileListFile

java.lang.Object

org.apache.hadoop.hbase.regionserver.storefiletracker.StoreFileListFile

@InterfaceAudience.Private
class StoreFileListFile
extends Object

To fully avoid listing, here we use two files for tracking. When loading, we will try to read both the two files, if only one exists, we will trust this one, if both exist, we will compare the timestamp to see which one is newer and trust that one. And we will record in memory that which one is trusted by us, and when we need to update the store file list, we will write to the other file.

So in this way, we could avoid listing when we want to load the store file list file.

To prevent loading partial file, we use the first 4 bytes as file length, and also append a 4 bytes crc32 checksum at the end. This is because the protobuf message parser sometimes can return without error on partial bytes if you stop at some special points, but the return message will have incorrect field value. We should try our best to prevent this happens because loading an incorrect store file list file usually leads to data loss.

Field Summary

Fields

Modifier and Type	Field and Description
private StoreContext	ctx
private static org.slf4j.Logger	LOG
private static int	MAX_FILE_SIZE
private int	nextTrackFile
private long	prevTimestamp
private static String	TRACK_FILE
(package private) static String	TRACK_FILE_DIR
private static String	TRACK_FILE_ROTATE
private org.apache.hadoop.fs.Path	trackFileDir
private org.apache.hadoop.fs.Path[]	trackFiles

Constructor Summary

Constructors

Constructor and Description
StoreFileListFile(StoreContext ctx)

Method Summary

Modifier and Type	Method and Description
(package private) org.apache.hadoop.hbase.shaded.protobuf.generated.StoreFileTrackerProtos.StoreFileList	load()
private org.apache.hadoop.hbase.shaded.protobuf.generated.StoreFileTrackerProtos.StoreFileList	load(org.apache.hadoop.fs.Path path)
private int	select(org.apache.hadoop.hbase.shaded.protobuf.generated.StoreFileTrackerProtos.StoreFileList[] lists)
(package private) void	update(org.apache.hadoop.hbase.shaded.protobuf.generated.StoreFileTrackerProtos.StoreFileList.Builder builder) We will set the timestamp in this method so just pass the builder in

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

private static final org.slf4j.Logger LOG

TRACK_FILE_DIR

static final String TRACK_FILE_DIR

See Also:

Constant Field Values

TRACK_FILE

private static final String TRACK_FILE

See Also:

Constant Field Values

TRACK_FILE_ROTATE

private static final String TRACK_FILE_ROTATE

See Also:

Constant Field Values

MAX_FILE_SIZE

private static final int MAX_FILE_SIZE

See Also:

Constant Field Values

ctx

private final StoreContext ctx

trackFileDir

private final org.apache.hadoop.fs.Path trackFileDir

trackFiles

private final org.apache.hadoop.fs.Path[] trackFiles

prevTimestamp

private long prevTimestamp

nextTrackFile

private int nextTrackFile

Constructor Detail

StoreFileListFile

StoreFileListFile(StoreContext ctx)

Method Detail

load

private org.apache.hadoop.hbase.shaded.protobuf.generated.StoreFileTrackerProtos.StoreFileList load(org.apache.hadoop.fs.Path path)
                                                                                             throws IOException

Throws:

IOException

select

private int select(org.apache.hadoop.hbase.shaded.protobuf.generated.StoreFileTrackerProtos.StoreFileList[] lists)

load

org.apache.hadoop.hbase.shaded.protobuf.generated.StoreFileTrackerProtos.StoreFileList load()
                                                                                     throws IOException

Throws:

IOException

update

void update(org.apache.hadoop.hbase.shaded.protobuf.generated.StoreFileTrackerProtos.StoreFileList.Builder builder)
     throws IOException

We will set the timestamp in this method so just pass the builder in

Throws:

IOException