org.apache.hadoop.hbase.io.hfile

Class ChecksumUtil

java.lang.Object

org.apache.hadoop.hbase.io.hfile.ChecksumUtil

@InterfaceAudience.Private
public class ChecksumUtil
extends Object

Utility methods to compute and validate checksums.

Field Summary

Fields

Modifier and Type	Field and Description
static int	CHECKSUM_BUF_SIZE
private static boolean	generateExceptions This is used by unit tests to make checksum failures throw an exception instead of returning null.
static org.slf4j.Logger	LOG

Constructor Summary

Constructors

Constructor and Description
ChecksumUtil()

Method Summary

Modifier and Type	Method and Description
(package private) static void	generateChecksums(byte[] indata, int startOffset, int endOffset, byte[] outdata, int outOffset, ChecksumType checksumType, int bytesPerChecksum) Generates a checksum for all the data in indata.
static void	generateExceptionForChecksumFailureForTest(boolean value) Mechanism to throw an exception in case of hbase checksum failure.
(package private) static long	numBytes(long datasize, int bytesPerChecksum) Returns the number of bytes needed to store the checksums for a specified data size
(package private) static long	numChunks(long datasize, int bytesPerChecksum) Returns the number of checksum chunks needed to store the checksums for a specified data size
(package private) static boolean	validateChecksum(ByteBuff buf, String pathName, long offset, int hdrSize) Validates that the data in the specified HFileBlock matches the checksum.
private static boolean	verifyChunkedSums(org.apache.hadoop.util.DataChecksum dataChecksum, ByteBuff data, ByteBuff checksums, String pathName) Like the hadoop's DataChecksum.verifyChunkedSums(ByteBuffer, ByteBuffer, String, long), this method will also verify checksum of each chunk in data.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.slf4j.Logger LOG

CHECKSUM_BUF_SIZE

public static final int CHECKSUM_BUF_SIZE

See Also:

Constant Field Values

generateExceptions

private static boolean generateExceptions

This is used by unit tests to make checksum failures throw an exception instead of returning null. Returning a null value from checksum validation will cause the higher layer to retry that read with hdfs-level checksums. Instead, we would like checksum failures to cause the entire unit test to fail.

Constructor Detail

ChecksumUtil

public ChecksumUtil()

Method Detail

generateChecksums

static void generateChecksums(byte[] indata,
                              int startOffset,
                              int endOffset,
                              byte[] outdata,
                              int outOffset,
                              ChecksumType checksumType,
                              int bytesPerChecksum)
                       throws IOException

Generates a checksum for all the data in indata. The checksum is written to outdata.

Parameters:

indata - input data stream

startOffset - starting offset in the indata stream from where to compute checkums from

endOffset - ending offset in the indata stream upto which checksums needs to be computed

outdata - the output buffer where checksum values are written

outOffset - the starting offset in the outdata where the checksum values are written

checksumType - type of checksum

bytesPerChecksum - number of bytes per checksum value

Throws:

IOException

verifyChunkedSums

private static boolean verifyChunkedSums(org.apache.hadoop.util.DataChecksum dataChecksum,
                                         ByteBuff data,
                                         ByteBuff checksums,
                                         String pathName)

Like the hadoop's DataChecksum.verifyChunkedSums(ByteBuffer, ByteBuffer, String, long), this method will also verify checksum of each chunk in data. the difference is: this method can accept ByteBuff as arguments, we can not add it in hadoop-common so defined here.

Parameters:

dataChecksum - to calculate the checksum.

data - as the input

checksums - to compare

pathName - indicate that the data is read from which file.

Returns:

a flag indicate the checksum match or mismatch.

See Also:

DataChecksum.verifyChunkedSums(ByteBuffer, ByteBuffer, String, long)

validateChecksum

static boolean validateChecksum(ByteBuff buf,
                                String pathName,
                                long offset,
                                int hdrSize)

Validates that the data in the specified HFileBlock matches the checksum. Generates the checksums for the data and then validate that it matches those stored in the end of the data.

Parameters:

buf - Contains the data in following order: HFileBlock header, data, checksums.

pathName - Path of the HFile to which the data belongs. Only used for logging.

offset - offset of the data being validated. Only used for logging.

hdrSize - Size of the block header in data. Only used for logging.

Returns:

True if checksum matches, else false.

numBytes

static long numBytes(long datasize,
                     int bytesPerChecksum)

Returns the number of bytes needed to store the checksums for a specified data size

Parameters:

datasize - number of bytes of data

bytesPerChecksum - number of bytes in a checksum chunk

Returns:

The number of bytes needed to store the checksum values

numChunks

static long numChunks(long datasize,
                      int bytesPerChecksum)

Returns the number of checksum chunks needed to store the checksums for a specified data size

Parameters:

datasize - number of bytes of data

bytesPerChecksum - number of bytes in a checksum chunk

Returns:

The number of checksum chunks

generateExceptionForChecksumFailureForTest

public static void generateExceptionForChecksumFailureForTest(boolean value)

Mechanism to throw an exception in case of hbase checksum failure. This is used by unit tests only.

Parameters:

value - Setting this to true will cause hbase checksum verification failures to generate exceptions.