org.apache.hadoop.hbase.regionserver.storefiletracker

Class StoreFileListFile

java.lang.Object

org.apache.hadoop.hbase.regionserver.storefiletracker.StoreFileListFile

@InterfaceAudience.Private
class StoreFileListFile
extends Object

To fully avoid listing, here we use two files for tracking. When loading, we will try to read both the two files, if only one exists, we will trust this one, if both exist, we will compare the timestamp to see which one is newer and trust that one. And we will record in memory that which one is trusted by us, and when we need to update the store file list, we will write to the other file.

So in this way, we could avoid listing when we want to load the store file list file.

To prevent loading partial file, we use the first 4 bytes as file length, and also append a 4 bytes crc32 checksum at the end. This is because the protobuf message parser sometimes can return without error on partial bytes if you stop at some special points, but the return message will have incorrect field value. We should try our best to prevent this happens because loading an incorrect store file list file usually leads to data loss.

Field Summary

Fields

Modifier and Type	Field and Description
private StoreContext	ctx
private static org.slf4j.Logger	LOG
private static int	MAX_FILE_SIZE
private int	nextTrackFile
private long	prevTimestamp
(package private) static String	TRACK_FILE_DIR
(package private) static Pattern	TRACK_FILE_PATTERN
private static String	TRACK_FILE_PREFIX
private static String	TRACK_FILE_ROTATE_PREFIX
private static char	TRACK_FILE_SEPARATOR
private org.apache.hadoop.fs.Path	trackFileDir
private org.apache.hadoop.fs.Path[]	trackFiles

Constructor Summary

Constructors

Constructor and Description
StoreFileListFile(StoreContext ctx)

Method Summary

Modifier and Type	Method and Description
private void	cleanUpTrackFiles(long loadedSeqId, NavigableMap<Long,List<org.apache.hadoop.fs.Path>> seqId2TrackFiles)
private void	initializeTrackFiles(long seqId)
private NavigableMap<Long,List<org.apache.hadoop.fs.Path>>	listFiles()
(package private) org.apache.hadoop.hbase.shaded.protobuf.generated.StoreFileTrackerProtos.StoreFileList	load(boolean readOnly)
(package private) static org.apache.hadoop.hbase.shaded.protobuf.generated.StoreFileTrackerProtos.StoreFileList	load(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path)
(package private) org.apache.hadoop.hbase.shaded.protobuf.generated.StoreFileTrackerProtos.StoreFileList	load(org.apache.hadoop.fs.Path path)
private int	select(org.apache.hadoop.hbase.shaded.protobuf.generated.StoreFileTrackerProtos.StoreFileList[] lists)
(package private) void	update(org.apache.hadoop.hbase.shaded.protobuf.generated.StoreFileTrackerProtos.StoreFileList.Builder builder) We will set the timestamp in this method so just pass the builder in

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

private static final org.slf4j.Logger LOG

TRACK_FILE_DIR

static final String TRACK_FILE_DIR

See Also:

Constant Field Values

TRACK_FILE_PREFIX

private static final String TRACK_FILE_PREFIX

See Also:

Constant Field Values

TRACK_FILE_ROTATE_PREFIX

private static final String TRACK_FILE_ROTATE_PREFIX

See Also:

Constant Field Values

TRACK_FILE_SEPARATOR

private static final char TRACK_FILE_SEPARATOR

See Also:

Constant Field Values

TRACK_FILE_PATTERN

static final Pattern TRACK_FILE_PATTERN

MAX_FILE_SIZE

private static final int MAX_FILE_SIZE

See Also:

Constant Field Values

ctx

private final StoreContext ctx

trackFileDir

private final org.apache.hadoop.fs.Path trackFileDir

trackFiles

private final org.apache.hadoop.fs.Path[] trackFiles

prevTimestamp

private long prevTimestamp

nextTrackFile

private int nextTrackFile

Constructor Detail

StoreFileListFile

StoreFileListFile(StoreContext ctx)

Method Detail

load

static org.apache.hadoop.hbase.shaded.protobuf.generated.StoreFileTrackerProtos.StoreFileList load(org.apache.hadoop.fs.FileSystem fs,
                                                                                                   org.apache.hadoop.fs.Path path)
                                                                                            throws IOException

Throws:

IOException

load

org.apache.hadoop.hbase.shaded.protobuf.generated.StoreFileTrackerProtos.StoreFileList load(org.apache.hadoop.fs.Path path)
                                                                                     throws IOException

Throws:

IOException

select

private int select(org.apache.hadoop.hbase.shaded.protobuf.generated.StoreFileTrackerProtos.StoreFileList[] lists)

listFiles

private NavigableMap<Long,List<org.apache.hadoop.fs.Path>> listFiles()
                                                              throws IOException

Throws:

IOException

initializeTrackFiles

private void initializeTrackFiles(long seqId)

cleanUpTrackFiles

private void cleanUpTrackFiles(long loadedSeqId,
                               NavigableMap<Long,List<org.apache.hadoop.fs.Path>> seqId2TrackFiles)

load

org.apache.hadoop.hbase.shaded.protobuf.generated.StoreFileTrackerProtos.StoreFileList load(boolean readOnly)
                                                                                     throws IOException

Throws:

IOException

update

void update(org.apache.hadoop.hbase.shaded.protobuf.generated.StoreFileTrackerProtos.StoreFileList.Builder builder)
     throws IOException

We will set the timestamp in this method so just pass the builder in

Throws:

IOException