org.apache.hadoop.hbase.io.hfile

Class CompoundBloomFilterWriter

java.lang.Object

org.apache.hadoop.hbase.io.hfile.CompoundBloomFilterBase

org.apache.hadoop.hbase.io.hfile.CompoundBloomFilterWriter

All Implemented Interfaces:

InlineBlockWriter, CellSink, ShipperListener, BloomFilterBase, BloomFilterWriter

@InterfaceAudience.Private
public class CompoundBloomFilterWriter
extends CompoundBloomFilterBase
implements BloomFilterWriter, InlineBlockWriter

Adds methods required for writing a compound Bloom filter to the data section of an HFile to the CompoundBloomFilter class.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
private class	CompoundBloomFilterWriter.MetaWriter
private static class	CompoundBloomFilterWriter.ReadyChunk A Bloom filter chunk enqueued for writing

Field Summary

Fields

Modifier and Type	Field and Description
private HFileBlockIndex.BlockIndexWriter	bloomBlockIndexWriter
private BloomType	bloomType
private boolean	cacheOnWrite Whether to cache-on-write compound Bloom filter chunks
private BloomFilterChunk	chunk The current chunk being written to
private int	chunkByteSize The size of individual Bloom filter chunks to create
private byte[]	firstKeyInChunk The first key in the current Bloom filter chunk.
private static org.slf4j.Logger	LOG
private int	maxFold Maximum fold factor
private Cell	prevCell The prev Cell that was processed
private BloomFilterChunk	prevChunk Previous chunk, so that we can create another similar chunk
private Queue<CompoundBloomFilterWriter.ReadyChunk>	readyChunks

Fields inherited from class org.apache.hadoop.hbase.io.hfile.CompoundBloomFilterBase

comparator, errorRate, hashType, numChunks, totalByteSize, totalKeyCount, totalMaxKeys, VERSION

Constructor Summary

Constructors

Constructor and Description
CompoundBloomFilterWriter(int chunkByteSizeHint, float errorRate, int hashType, int maxFold, boolean cacheOnWrite, CellComparator comparator, BloomType bloomType)

Method Summary

Modifier and Type	Method and Description
private void	allocateNewChunk()
void	append(Cell cell) Append the given cell
void	beforeShipped() The action that needs to be performed before Shipper.shipped() is performed
void	blockWritten(long offset, int onDiskSize, int uncompressedSize) Called after a block has been written, and its offset, raw size, and compressed size have been determined.
void	compactBloom() Compact the Bloom filter before writing metadata & data to disk.
private void	enqueueReadyChunk(boolean closing) Enqueue the current chunk if it is ready to be written out.
boolean	getCacheOnWrite()
org.apache.hadoop.io.Writable	getDataWriter() Get a writable interface into bloom filter data (the actual Bloom bits).
BlockType	getInlineBlockType() The type of blocks this block writer produces.
org.apache.hadoop.io.Writable	getMetaWriter() Get a writable interface into bloom filter meta data.
Cell	getPrevCell() Returns the previous cell written by this writer
boolean	shouldWriteBlock(boolean closing) Determines whether there is a new block to be written out.
void	writeInlineBlock(DataOutput out) Writes the block to the provided stream.

Methods inherited from class org.apache.hadoop.hbase.io.hfile.CompoundBloomFilterBase

getByteSize, getKeyCount, getMaxKeys

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.hbase.util.BloomFilterBase

getByteSize, getKeyCount, getMaxKeys

Field Detail

LOG

private static final org.slf4j.Logger LOG

chunk

private BloomFilterChunk chunk

The current chunk being written to

prevChunk

private BloomFilterChunk prevChunk

Previous chunk, so that we can create another similar chunk

maxFold

private int maxFold

Maximum fold factor

chunkByteSize

private int chunkByteSize

The size of individual Bloom filter chunks to create

prevCell

private Cell prevCell

The prev Cell that was processed

readyChunks

private Queue<CompoundBloomFilterWriter.ReadyChunk> readyChunks

firstKeyInChunk

private byte[] firstKeyInChunk

The first key in the current Bloom filter chunk.

bloomBlockIndexWriter

private HFileBlockIndex.BlockIndexWriter bloomBlockIndexWriter

cacheOnWrite

private boolean cacheOnWrite

Whether to cache-on-write compound Bloom filter chunks

bloomType

private BloomType bloomType

Constructor Detail

CompoundBloomFilterWriter

public CompoundBloomFilterWriter(int chunkByteSizeHint,
                                 float errorRate,
                                 int hashType,
                                 int maxFold,
                                 boolean cacheOnWrite,
                                 CellComparator comparator,
                                 BloomType bloomType)

Parameters:

chunkByteSizeHint - each chunk's size in bytes. The real chunk size might be different as required by the fold factor.

errorRate - target false positive rate

hashType - hash function type to use

maxFold - maximum degree of folding allowed

bloomType - the bloom type

Method Detail

shouldWriteBlock

public boolean shouldWriteBlock(boolean closing)

Description copied from interface: InlineBlockWriter

Determines whether there is a new block to be written out.

Specified by:

shouldWriteBlock in interface InlineBlockWriter

Parameters:

closing - whether the file is being closed, in which case we need to write out all available data and not wait to accumulate another block

enqueueReadyChunk

private void enqueueReadyChunk(boolean closing)

Enqueue the current chunk if it is ready to be written out.

Parameters:

closing - true if we are closing the file, so we do not expect new keys to show up

append

public void append(Cell cell)
            throws IOException

Description copied from interface: CellSink

Append the given cell

Specified by:

append in interface CellSink

Parameters:

cell - the cell to be added

Throws:

IOException

beforeShipped

public void beforeShipped()
                   throws IOException

Description copied from interface: ShipperListener

The action that needs to be performed before Shipper.shipped() is performed

Specified by:

beforeShipped in interface ShipperListener

Throws:

IOException

getPrevCell

public Cell getPrevCell()

Description copied from interface: BloomFilterWriter

Returns the previous cell written by this writer

Specified by:

getPrevCell in interface BloomFilterWriter

Returns:

the previous cell

allocateNewChunk

private void allocateNewChunk()

writeInlineBlock

public void writeInlineBlock(DataOutput out)
                      throws IOException

Description copied from interface: InlineBlockWriter

Writes the block to the provided stream. Must not write any magic records. Called only if InlineBlockWriter.shouldWriteBlock(boolean) returned true.

Specified by:

writeInlineBlock in interface InlineBlockWriter

Parameters:

out - a stream (usually a compressing stream) to write the block to

Throws:

IOException

blockWritten

public void blockWritten(long offset,
                         int onDiskSize,
                         int uncompressedSize)

Description copied from interface: InlineBlockWriter

Called after a block has been written, and its offset, raw size, and compressed size have been determined. Can be used to add an entry to a block index. If this type of inline blocks needs a block index, the inline block writer is responsible for maintaining it.

Specified by:

blockWritten in interface InlineBlockWriter

Parameters:

offset - the offset of the block in the stream

onDiskSize - the on-disk size of the block

uncompressedSize - the uncompressed size of the block

getInlineBlockType

public BlockType getInlineBlockType()

Description copied from interface: InlineBlockWriter

The type of blocks this block writer produces.

Specified by:

getInlineBlockType in interface InlineBlockWriter

compactBloom

public void compactBloom()

Description copied from interface: BloomFilterWriter

Compact the Bloom filter before writing metadata & data to disk.

Specified by:

compactBloom in interface BloomFilterWriter

getMetaWriter

public org.apache.hadoop.io.Writable getMetaWriter()

Description copied from interface: BloomFilterWriter

Get a writable interface into bloom filter meta data.

Specified by:

getMetaWriter in interface BloomFilterWriter

Returns:

a writable instance that can be later written to a stream

getDataWriter

public org.apache.hadoop.io.Writable getDataWriter()

Description copied from interface: BloomFilterWriter

Get a writable interface into bloom filter data (the actual Bloom bits). Not used for compound Bloom filters.

Specified by:

getDataWriter in interface BloomFilterWriter

Returns:

a writable instance that can be later written to a stream

getCacheOnWrite

public boolean getCacheOnWrite()

Specified by:

getCacheOnWrite in interface InlineBlockWriter

Returns:

true if inline blocks produced by this writer should be cached