Field Detail

• LOG

  public static final org.apache.commons.logging.Log LOG

• DUMMY_VALUE

  private static byte[] DUMMY_VALUE

  This is used to reserve space in a byte buffer

• generateExceptions

  private static boolean generateExceptions

  This is used by unit tests to make checksum failures throw an exception instead of returning null. Returning a null value from checksum validation will cause the higher layer to retry that read with hdfs-level checksums. Instead, we would like checksum failures to cause the entire unit test to fail.

Constructor Detail

• ChecksumUtil

  public ChecksumUtil()

Method Detail

• generateChecksums

  static void generateChecksums(byte[] indata,
                       int startOffset,
                       int endOffset,
                       byte[] outdata,
                       int outOffset,
                       ChecksumType checksumType,
                       int bytesPerChecksum)
                         throws IOException

  Generates a checksum for all the data in indata. The checksum is written to outdata.

  Parameters:

  indata - input data stream

  startOffset - starting offset in the indata stream from where to compute checkums from

  endOffset - ending offset in the indata stream upto which checksums needs to be computed

  outdata - the output buffer where checksum values are written

  outOffset - the starting offset in the outdata where the checksum values are written

  checksumType - type of checksum

  bytesPerChecksum - number of bytes per checksum value

  Throws:

  IOException

• validateChecksum

  static boolean validateChecksum(ByteBuffer buffer,
                         org.apache.hadoop.fs.Path path,
                         long offset,
                         int hdrSize)
                           throws IOException

  Validates that the data in the specified HFileBlock matches the checksum. Generates the checksums for the data and then validate that it matches those stored in the end of the data.

  Parameters:

  buffer - Contains the data in following order: HFileBlock header, data, checksums.

  path - Path of the HFile to which the data belongs. Only used for logging.

  offset - offset of the data being validated. Only used for logging.

  hdrSize - Size of the block header in data. Only used for logging.

  Returns:

  True if checksum matches, else false.

  Throws:

  IOException

• numBytes

  static long numBytes(long datasize,
              int bytesPerChecksum)

  Returns the number of bytes needed to store the checksums for a specified data size

  Parameters:

  datasize - number of bytes of data

  bytesPerChecksum - number of bytes in a checksum chunk

  Returns:

  The number of bytes needed to store the checksum values

• numChunks

  static long numChunks(long datasize,
               int bytesPerChecksum)

  Returns the number of checksum chunks needed to store the checksums for a specified data size

  Parameters:

  datasize - number of bytes of data

  bytesPerChecksum - number of bytes in a checksum chunk

  Returns:

  The number of checksum chunks

• reserveSpaceForChecksums

  static void reserveSpaceForChecksums(ByteArrayOutputStream baos,
                              int numBytes,
                              int bytesPerChecksum)
                                throws IOException

  Write dummy checksums to the end of the specified bytes array to reserve space for writing checksums later

  Parameters:

  baos - OutputStream to write dummy checkum values

  numBytes - Number of bytes of data for which dummy checksums need to be generated

  bytesPerChecksum - Number of bytes per checksum value

  Throws:

  IOException

• generateExceptionForChecksumFailureForTest

  public static void generateExceptionForChecksumFailureForTest(boolean value)

  Mechanism to throw an exception in case of hbase checksum failure. This is used by unit tests only.

  Parameters:

  value - Setting this to true will cause hbase checksum verification failures to generate exceptions.