@InterfaceAudience.Private public class FastDiffDeltaEncoder extends BufferedDataBlockEncoder
DiffKeyDeltaEncoder
but supposedly faster. Compress using: - store
size of common prefix - save column family once in the first KeyValue - use integer compression
for key, value and prefix (7-bit encoding) - use bits to avoid duplication key length, value
length and type if it same as previous - store in 3 bits length of prefix timestamp with previous
KeyValue's timestamp - one bit which allow to omit value if it is the same Format: - 1 byte: flag
- 1-5 bytes: key length (only if FLAG_SAME_KEY_LENGTH is not set in flag) - 1-5 bytes: value
length (only if FLAG_SAME_VALUE_LENGTH is not set in flag) - 1-5 bytes: prefix length - ...
bytes: rest of the row (if prefix length is small enough) - ... bytes: qualifier (or suffix
depending on prefix length) - 1-8 bytes: timestamp suffix - 1 byte: type (only if FLAG_SAME_TYPE
is not set in the flag) - ... bytes: value (only if FLAG_SAME_VALUE is not set in the flag)Modifier and Type | Class and Description |
---|---|
private static class |
FastDiffDeltaEncoder.FastDiffCompressionState |
protected static class |
FastDiffDeltaEncoder.FastDiffSeekerState |
private static class |
FastDiffDeltaEncoder.FastDiffSeekerStateBufferedEncodedSeeker |
BufferedDataBlockEncoder.BufferedEncodedSeeker<STATE extends BufferedDataBlockEncoder.SeekerState>, BufferedDataBlockEncoder.OffheapDecodedExtendedCell, BufferedDataBlockEncoder.OnheapDecodedCell, BufferedDataBlockEncoder.SeekerState
AbstractDataBlockEncoder.AbstractEncodedSeeker
DataBlockEncoder.EncodedSeeker
Modifier and Type | Field and Description |
---|---|
(package private) static int |
FLAG_SAME_KEY_LENGTH |
(package private) static int |
FLAG_SAME_TYPE |
(package private) static int |
FLAG_SAME_VALUE |
(package private) static int |
FLAG_SAME_VALUE_LENGTH |
(package private) static int |
MASK_TIMESTAMP_LENGTH |
(package private) static int |
SHIFT_TIMESTAMP_LENGTH |
Constructor and Description |
---|
FastDiffDeltaEncoder() |
Modifier and Type | Method and Description |
---|---|
private int |
compressSingleKeyValue(DataOutputStream out,
Cell cell,
Cell prevCell) |
DataBlockEncoder.EncodedSeeker |
createSeeker(HFileBlockDecodingContext decodingCtx)
Create a HFileBlock seeker which find KeyValues within a block.
|
private int |
findCommonTimestampPrefix(byte[] curTsBuf,
byte[] prevTsBuf) |
Cell |
getFirstKeyCellInBlock(ByteBuff block)
Return first key in block as a cell.
|
protected ByteBuffer |
internalDecodeKeyValues(DataInputStream source,
int allocateHeaderLength,
int skipLastBytes,
HFileBlockDefaultDecodingContext decodingCtx) |
int |
internalEncode(Cell cell,
HFileBlockDefaultEncodingContext encodingContext,
DataOutputStream out) |
String |
toString() |
private void |
uncompressSingleKeyValue(DataInputStream source,
ByteBuffer out,
FastDiffDeltaEncoder.FastDiffCompressionState state) |
afterDecodingKeyValue, afterEncodingKeyValue, compareCommonFamilyPrefix, compareCommonQualifierPrefix, compareCommonRowPrefix, decodeKeyValues, encode, endBlockEncoding, ensureSpace, startBlockEncoding
createFirstKeyCell, newDataBlockDecodingContext, newDataBlockEncodingContext, postEncoding
static final int MASK_TIMESTAMP_LENGTH
static final int SHIFT_TIMESTAMP_LENGTH
static final int FLAG_SAME_KEY_LENGTH
static final int FLAG_SAME_VALUE_LENGTH
static final int FLAG_SAME_TYPE
static final int FLAG_SAME_VALUE
public FastDiffDeltaEncoder()
private int findCommonTimestampPrefix(byte[] curTsBuf, byte[] prevTsBuf)
private void uncompressSingleKeyValue(DataInputStream source, ByteBuffer out, FastDiffDeltaEncoder.FastDiffCompressionState state) throws IOException, EncoderBufferTooSmallException
public int internalEncode(Cell cell, HFileBlockDefaultEncodingContext encodingContext, DataOutputStream out) throws IOException
internalEncode
in class BufferedDataBlockEncoder
IOException
private int compressSingleKeyValue(DataOutputStream out, Cell cell, Cell prevCell) throws IOException
IOException
protected ByteBuffer internalDecodeKeyValues(DataInputStream source, int allocateHeaderLength, int skipLastBytes, HFileBlockDefaultDecodingContext decodingCtx) throws IOException
internalDecodeKeyValues
in class BufferedDataBlockEncoder
IOException
public Cell getFirstKeyCellInBlock(ByteBuff block)
DataBlockEncoder
block
- encoded block we want index, the position will not changepublic DataBlockEncoder.EncodedSeeker createSeeker(HFileBlockDecodingContext decodingCtx)
DataBlockEncoder
Copyright © 2007–2020 The Apache Software Foundation. All rights reserved.