org.apache.hadoop.hbase.io.hfile

Class ChecksumUtil

java.lang.Object

org.apache.hadoop.hbase.io.hfile.ChecksumUtil

@InterfaceAudience.Private
public class ChecksumUtil
extends Object

Utility methods to compute and validate checksums.

Field Summary

Fields

Modifier and Type	Field and Description
private static byte[]	DUMMY_VALUE This is used to reserve space in a byte buffer
private static boolean	generateExceptions This is used by unit tests to make checksum failures throw an exception instead of returning null.
static org.slf4j.Logger	LOG

Constructor Summary

Constructors

Constructor and Description
ChecksumUtil()

Method Summary

Modifier and Type	Method and Description
(package private) static void	generateChecksums(byte[] indata, int startOffset, int endOffset, byte[] outdata, int outOffset, ChecksumType checksumType, int bytesPerChecksum) Generates a checksum for all the data in indata.
static void	generateExceptionForChecksumFailureForTest(boolean value) Mechanism to throw an exception in case of hbase checksum failure.
(package private) static long	numBytes(long datasize, int bytesPerChecksum) Returns the number of bytes needed to store the checksums for a specified data size
(package private) static long	numChunks(long datasize, int bytesPerChecksum) Returns the number of checksum chunks needed to store the checksums for a specified data size
(package private) static void	reserveSpaceForChecksums(ByteArrayOutputStream baos, int numBytes, int bytesPerChecksum) Write dummy checksums to the end of the specified bytes array to reserve space for writing checksums later
(package private) static boolean	validateChecksum(ByteBuffer buffer, String pathName, long offset, int hdrSize) Validates that the data in the specified HFileBlock matches the checksum.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.slf4j.Logger LOG

DUMMY_VALUE

private static byte[] DUMMY_VALUE

This is used to reserve space in a byte buffer

generateExceptions

private static boolean generateExceptions

This is used by unit tests to make checksum failures throw an exception instead of returning null. Returning a null value from checksum validation will cause the higher layer to retry that read with hdfs-level checksums. Instead, we would like checksum failures to cause the entire unit test to fail.

Constructor Detail

ChecksumUtil

public ChecksumUtil()

Method Detail

generateChecksums

static void generateChecksums(byte[] indata,
                              int startOffset,
                              int endOffset,
                              byte[] outdata,
                              int outOffset,
                              ChecksumType checksumType,
                              int bytesPerChecksum)
                       throws IOException

Generates a checksum for all the data in indata. The checksum is written to outdata.

Parameters:

indata - input data stream

startOffset - starting offset in the indata stream from where to compute checkums from

endOffset - ending offset in the indata stream upto which checksums needs to be computed

outdata - the output buffer where checksum values are written

outOffset - the starting offset in the outdata where the checksum values are written

checksumType - type of checksum

bytesPerChecksum - number of bytes per checksum value

Throws:

IOException

validateChecksum

static boolean validateChecksum(ByteBuffer buffer,
                                String pathName,
                                long offset,
                                int hdrSize)
                         throws IOException

Validates that the data in the specified HFileBlock matches the checksum. Generates the checksums for the data and then validate that it matches those stored in the end of the data.

Parameters:

buffer - Contains the data in following order: HFileBlock header, data, checksums.

pathName - Path of the HFile to which the data belongs. Only used for logging.

offset - offset of the data being validated. Only used for logging.

hdrSize - Size of the block header in data. Only used for logging.

Returns:

True if checksum matches, else false.

Throws:

IOException

numBytes

static long numBytes(long datasize,
                     int bytesPerChecksum)

Returns the number of bytes needed to store the checksums for a specified data size

Parameters:

datasize - number of bytes of data

bytesPerChecksum - number of bytes in a checksum chunk

Returns:

The number of bytes needed to store the checksum values

numChunks

static long numChunks(long datasize,
                      int bytesPerChecksum)

Returns the number of checksum chunks needed to store the checksums for a specified data size

Parameters:

datasize - number of bytes of data

bytesPerChecksum - number of bytes in a checksum chunk

Returns:

The number of checksum chunks

reserveSpaceForChecksums

static void reserveSpaceForChecksums(ByteArrayOutputStream baos,
                                     int numBytes,
                                     int bytesPerChecksum)
                              throws IOException

Write dummy checksums to the end of the specified bytes array to reserve space for writing checksums later

Parameters:

baos - OutputStream to write dummy checkum values

numBytes - Number of bytes of data for which dummy checksums need to be generated

bytesPerChecksum - Number of bytes per checksum value

Throws:

IOException

generateExceptionForChecksumFailureForTest

public static void generateExceptionForChecksumFailureForTest(boolean value)

Mechanism to throw an exception in case of hbase checksum failure. This is used by unit tests only.

Parameters:

value - Setting this to true will cause hbase checksum verification failures to generate exceptions.