Package org.apache.hadoop.hbase.regionserver.wal

Class WALUtil

java.lang.Object

org.apache.hadoop.hbase.regionserver.wal.WALUtil

@Private
public class WALUtil
extends Object

Helper methods to ease Region Server integration with the Write Ahead Log (WAL). Note that methods in this class specifically should not require access to anything other than the API found in WAL. For internal use only.

Field Summary

Fields

Modifier and Type	Field	Description
private static final org.slf4j.Logger	LOG
static final String	WAL_BLOCK_SIZE

Constructor Summary

Constructors

Modifier	Constructor	Description
private	WALUtil()

Method Summary

Modifier and Type	Method	Description
static WALKeyImpl	createWALKey(RegionInfo hri, MultiVersionConcurrencyControl mvcc, NavigableMap<byte[],Integer> replicationScope, Map<String,byte[]> extendedAttributes)
private static WALKeyImpl	doFullMarkerAppendTransaction(WAL wal, NavigableMap<byte[],Integer> replicationScope, RegionInfo hri, WALEdit edit, MultiVersionConcurrencyControl mvcc, Map<String,byte[]> extendedAttributes, boolean sync, RegionReplicationSink sink)	A 'full' WAL transaction involves starting an mvcc transaction followed by an append, an optional sync, and then a call to complete the mvcc transaction.
static void	filterCells(WALEdit edit, Function<Cell,Cell> mapper)
static long	getWALBlockSize(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path dir)	Blocksize returned here is 2x the default HDFS blocksize unless explicitly set in Configuration.
static long	getWALBlockSize(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path dir, boolean isRecoverEdits)	Public because of FSHLog.
static WALKeyImpl	writeBulkLoadMarkerAndSync(WAL wal, NavigableMap<byte[],Integer> replicationScope, RegionInfo hri, org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos.BulkLoadDescriptor desc, MultiVersionConcurrencyControl mvcc, RegionReplicationSink sink)	Write a log marker that a bulk load has succeeded and is about to be committed.
static WALKeyImpl	writeCompactionMarker(WAL wal, NavigableMap<byte[],Integer> replicationScope, RegionInfo hri, org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos.CompactionDescriptor c, MultiVersionConcurrencyControl mvcc, RegionReplicationSink sink)	Write the marker that a compaction has succeeded and is about to be committed.
static WALKeyImpl	writeFlushMarker(WAL wal, NavigableMap<byte[],Integer> replicationScope, RegionInfo hri, org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos.FlushDescriptor f, boolean sync, MultiVersionConcurrencyControl mvcc, RegionReplicationSink sink)	Write a flush marker indicating a start / abort or a complete of a region flush
private static WALKeyImpl	writeMarker(WAL wal, NavigableMap<byte[],Integer> replicationScope, RegionInfo hri, WALEdit edit, MultiVersionConcurrencyControl mvcc, Map<String,byte[]> extendedAttributes, RegionReplicationSink sink)
static WALKeyImpl	writeRegionEventMarker(WAL wal, NavigableMap<byte[],Integer> replicationScope, RegionInfo hri, org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos.RegionEventDescriptor r, MultiVersionConcurrencyControl mvcc, RegionReplicationSink sink)	Write a region open marker indicating that the region is opened.
static void	writeReplicationMarkerAndSync(WAL wal, MultiVersionConcurrencyControl mvcc, RegionInfo regionInfo, byte[] rowKey, long timestamp)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

LOG

private static final org.slf4j.Logger LOG

WAL_BLOCK_SIZE

public static final String WAL_BLOCK_SIZE

See Also:

• Constant Field Values

Constructor Details

WALUtil

private WALUtil()

Method Details

writeCompactionMarker

public static WALKeyImpl writeCompactionMarker(WAL wal,
 NavigableMap<byte[],Integer> replicationScope,
 RegionInfo hri,
 org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos.CompactionDescriptor c,
 MultiVersionConcurrencyControl mvcc,
 RegionReplicationSink sink)
                                        throws IOException

Write the marker that a compaction has succeeded and is about to be committed. This provides info to the HMaster to allow it to recover the compaction if this regionserver dies in the middle. It also prevents the compaction from finishing if this regionserver has already lost its lease on the log.

This write is for internal use only. Not for external client consumption.

Parameters:

mvcc - Used by WAL to get sequence Id for the waledit.

Throws:

IOException

writeFlushMarker

public static WALKeyImpl writeFlushMarker(WAL wal,
 NavigableMap<byte[],Integer> replicationScope,
 RegionInfo hri,
 org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos.FlushDescriptor f,
 boolean sync,
 MultiVersionConcurrencyControl mvcc,
 RegionReplicationSink sink)
                                   throws IOException

Write a flush marker indicating a start / abort or a complete of a region flush

This write is for internal use only. Not for external client consumption.

Throws:

IOException

writeRegionEventMarker

public static WALKeyImpl writeRegionEventMarker(WAL wal,
 NavigableMap<byte[],Integer> replicationScope,
 RegionInfo hri,
 org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos.RegionEventDescriptor r,
 MultiVersionConcurrencyControl mvcc,
 RegionReplicationSink sink)
                                         throws IOException

Write a region open marker indicating that the region is opened. This write is for internal use only. Not for external client consumption.

Throws:

IOException

writeBulkLoadMarkerAndSync

public static WALKeyImpl writeBulkLoadMarkerAndSync(WAL wal,
 NavigableMap<byte[],Integer> replicationScope,
 RegionInfo hri,
 org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos.BulkLoadDescriptor desc,
 MultiVersionConcurrencyControl mvcc,
 RegionReplicationSink sink)
                                             throws IOException

Write a log marker that a bulk load has succeeded and is about to be committed. This write is for internal use only. Not for external client consumption.

Parameters:

wal - The log to write into.

replicationScope - The replication scope of the families in the HRegion

hri - A description of the region in the table that we are bulk loading into.

desc - A protocol buffers based description of the client's bulk loading request

Returns:

walKey with sequenceid filled out for this bulk load marker

Throws:

IOException - We will throw an IOException if we can not append to the HLog.

writeMarker

private static WALKeyImpl writeMarker(WAL wal,
 NavigableMap<byte[],Integer> replicationScope,
 RegionInfo hri,
 WALEdit edit,
 MultiVersionConcurrencyControl mvcc,
 Map<String,byte[]> extendedAttributes,
 RegionReplicationSink sink)
                               throws IOException

Throws:

IOException

doFullMarkerAppendTransaction

private static WALKeyImpl doFullMarkerAppendTransaction(WAL wal,
 NavigableMap<byte[],Integer> replicationScope,
 RegionInfo hri,
 WALEdit edit,
 MultiVersionConcurrencyControl mvcc,
 Map<String,byte[]> extendedAttributes,
 boolean sync,
 RegionReplicationSink sink)
                                                 throws IOException

A 'full' WAL transaction involves starting an mvcc transaction followed by an append, an optional sync, and then a call to complete the mvcc transaction. This method does it all. Good for case of adding a single edit or marker to the WAL.

This write is for internal use only. Not for external client consumption.

Returns:

WALKeyImpl that was added to the WAL.

Throws:

IOException

createWALKey

public static WALKeyImpl createWALKey(RegionInfo hri,
 MultiVersionConcurrencyControl mvcc,
 NavigableMap<byte[],Integer> replicationScope,
 Map<String,byte[]> extendedAttributes)

getWALBlockSize

public static long getWALBlockSize(org.apache.hadoop.conf.Configuration conf,
 org.apache.hadoop.fs.FileSystem fs,
 org.apache.hadoop.fs.Path dir)
                            throws IOException

Blocksize returned here is 2x the default HDFS blocksize unless explicitly set in Configuration. Works in tandem with hbase.regionserver.logroll.multiplier. See comment in AbstractFSWAL in Constructor where we set blocksize and logrollsize for why.

Returns:

Blocksize to use writing WALs.

Throws:

IOException

getWALBlockSize

public static long getWALBlockSize(org.apache.hadoop.conf.Configuration conf,
 org.apache.hadoop.fs.FileSystem fs,
 org.apache.hadoop.fs.Path dir,
 boolean isRecoverEdits)
                            throws IOException

Public because of FSHLog. Should be package-private

Parameters:

isRecoverEdits - the created writer is for recovered edits or WAL. For recovered edits, it is true and for WAL it is false.

Throws:

IOException

filterCells

public static void filterCells(WALEdit edit,
 Function<Cell,Cell> mapper)

writeReplicationMarkerAndSync

public static void writeReplicationMarkerAndSync(WAL wal,
 MultiVersionConcurrencyControl mvcc,
 RegionInfo regionInfo,
 byte[] rowKey,
 long timestamp)
                                          throws IOException

Throws:

IOException