Package org.apache.hadoop.hbase.io.hfile.bucket

Class ByteBufferIOEngine

java.lang.Object
org.apache.hadoop.hbase.io.hfile.bucket.ByteBufferIOEngine
All Implemented Interfaces:
IOEngine
@Private public class ByteBufferIOEngine extends Object implements IOEngine
IO engine that stores data in memory using an array of ByteBuffers ByteBufferArray.

How it Works

First, see ByteBufferArray and how it gives a view across multiple ByteBuffers managed by it internally. This class does the physical BB create and the write and read to the underlying BBs. So we will create N BBs based on the total BC capacity specified on create of the ByteBufferArray. So say we have 10 GB of off heap BucketCache, we will create 2560 such BBs inside our ByteBufferArray.

Now the way BucketCache works is that the entire 10 GB is split into diff sized buckets: by default from 5 KB to 513 KB. Within each bucket of a particular size, there are usually more than one bucket 'block'. The way it is calculate in bucketcache is that the total bucketcache size is divided by 4 (hard-coded currently) * max size option. So using defaults, buckets will be is 4 * 513kb (the biggest default value) = 2052kb. A bucket of 2052kb at offset zero will serve out bucket 'blocks' of 5kb, the next bucket will do the next size up and so on up to the maximum (default) of 513kb).

When we write blocks to the bucketcache, we will see which bucket size group it best fits. So a 4 KB block size goes to the 5 KB size group. Each of the block writes, writes within its appropriate bucket. Though the bucket is '4kb' in size, it will occupy one of the 5 KB bucket 'blocks' (even if actual size of the bucket is less). Bucket 'blocks' will not span buckets.

But you can see the physical memory under the bucket 'blocks' can be split across the underlying backing BBs from ByteBufferArray. All is split into 4 MB sized BBs.

Each Bucket knows its offset in the entire space of BC and when block is written the offset arrives at ByteBufferArray and it figures which BB to write to. It may so happen that the entire block to be written does not fit a particular backing ByteBufferArray so the remainder goes to another BB. See ByteBufferArray.write(long, ByteBuff).
So said all these, when we read a block it may be possible that the bytes of that blocks is physically placed in 2 adjucent BBs. In such case also, we avoid any copy need by having the MBB...

  • Field Details

  • Constructor Details

    • ByteBufferIOEngine

      public ByteBufferIOEngine(long capacity) throws IOException
      Construct the ByteBufferIOEngine with the given capacity
      Throws:
      IOException - ideally here no exception to be thrown from the allocator

  • Method Details

    • toString

      public String toString()
      Overrides:
      toString in class Object

    • isPersistent

      public boolean isPersistent()
      Memory IO engine is always unable to support persistent storage for the cache
      Specified by:
      isPersistent in interface IOEngine

    • usesSharedMemory

      public boolean usesSharedMemory()
      Description copied from interface: IOEngine
      IOEngine uses shared memory means, when reading Cacheable from it, those refers to the same memory area as used by the Engine for caching it.
      Specified by:
      usesSharedMemory in interface IOEngine
      Returns:
      true when IOEngine using shared memory.

    • read

      public Cacheable read(BucketEntry be) throws IOException
      Description copied from interface: IOEngine
      Transfers data from IOEngine to a Cacheable object.
      Specified by:
      read in interface IOEngine
      Parameters:
      be - maintains an (offset,len,refCnt) inside.
      Returns:
      Cacheable which will wrap the NIO ByteBuffers from IOEngine.
      Throws:
      IOException - when any IO error happen

    • write

      public void write(ByteBuffer src, long offset) throws IOException
      Transfers data from the given ByteBuffer to the buffer array. Position of source will be advanced by the Buffer.remaining().
      Specified by:
      write in interface IOEngine
      Parameters:
      src - the given byte buffer from which bytes are to be read.
      offset - The offset in the ByteBufferArray of the first byte to be written
      Throws:
      IOException - throws IOException if writing to the array throws exception

    • write

      public void write(ByteBuff src, long offset) throws IOException
      Transfers data from the given ByteBuff to the buffer array. Position of source will be advanced by the Buffer.remaining().
      Specified by:
      write in interface IOEngine
      Parameters:
      src - the given byte buffer from which bytes are to be read.
      offset - The offset in the ByteBufferArray of the first byte to be written
      Throws:
      IOException - throws IOException if writing to the array throws exception

    • sync

      public void sync()
      No operation for the sync in the memory IO engine
      Specified by:
      sync in interface IOEngine

    • shutdown

      public void shutdown()
      No operation for the shutdown in the memory IO engine
      Specified by:
      shutdown in interface IOEngine