@InterfaceAudience.Private public class ByteBufferIOEngine extends Object implements IOEngine
ByteBufferArray.
ByteBufferArray and how it gives a view across multiple ByteBuffers managed
by it internally. This class does the physical BB create and the write and read to the
underlying BBs. So we will create N BBs based on the total BC capacity specified on create
of the ByteBufferArray. So say we have 10 GB of off heap BucketCache, we will create 2560 such
BBs inside our ByteBufferArray.
Now the way BucketCache works is that the entire 10 GB is split into diff sized buckets: by default from 5 KB to 513 KB. Within each bucket of a particular size, there are usually more than one bucket 'block'. The way it is calculate in bucketcache is that the total bucketcache size is divided by 4 (hard-coded currently) * max size option. So using defaults, buckets will be is 4 * 513kb (the biggest default value) = 2052kb. A bucket of 2052kb at offset zero will serve out bucket 'blocks' of 5kb, the next bucket will do the next size up and so on up to the maximum (default) of 513kb).
When we write blocks to the bucketcache, we will see which bucket size group it best fits. So a 4 KB block size goes to the 5 KB size group. Each of the block writes, writes within its appropriate bucket. Though the bucket is '4kb' in size, it will occupy one of the 5 KB bucket 'blocks' (even if actual size of the bucket is less). Bucket 'blocks' will not span buckets.
But you can see the physical memory under the bucket 'blocks' can be split across the underlying backing BBs from ByteBufferArray. All is split into 4 MB sized BBs.
Each Bucket knows its offset in the entire space of BC and when block is written the offset
arrives at ByteBufferArray and it figures which BB to write to. It may so happen that the entire
block to be written does not fit a particular backing ByteBufferArray so the remainder goes to
another BB. See ByteBufferArray.putMultiple(long, int, byte[]).
So said all these, when we read a block it may be possible that the bytes of that blocks is physically placed in 2 adjucent BBs. In such case also, we avoid any copy need by having the MBB...
| Modifier and Type | Field and Description |
|---|---|
private ByteBufferArray |
bufferArray |
private long |
capacity |
| Constructor and Description |
|---|
ByteBufferIOEngine(long capacity)
Construct the ByteBufferIOEngine with the given capacity
|
| Modifier and Type | Method and Description |
|---|---|
boolean |
isPersistent()
Memory IO engine is always unable to support persistent storage for the
cache
|
Cacheable |
read(long offset,
int length,
CacheableDeserializer<Cacheable> deserializer)
Transfers data from IOEngine to a Cacheable object.
|
void |
shutdown()
No operation for the shutdown in the memory IO engine
|
void |
sync()
No operation for the sync in the memory IO engine
|
String |
toString() |
boolean |
usesSharedMemory()
IOEngine uses shared memory means, when reading Cacheable from it, those refers to the same
memory area as used by the Engine for caching it.
|
void |
write(ByteBuffer srcBuffer,
long offset)
Transfers data from the given byte buffer to the buffer array
|
void |
write(ByteBuff srcBuffer,
long offset)
Transfers the data from the given MultiByteBuffer to IOEngine
|
private ByteBufferArray bufferArray
private final long capacity
public ByteBufferIOEngine(long capacity) throws IOException
capacity - IOException - ideally here no exception to be thrown from the allocatorpublic boolean isPersistent()
isPersistent in interface IOEnginepublic boolean usesSharedMemory()
IOEngineusesSharedMemory in interface IOEnginepublic Cacheable read(long offset, int length, CacheableDeserializer<Cacheable> deserializer) throws IOException
IOEngineread in interface IOEngineoffset - The offset in the IO engine where the first byte to be readlength - How many bytes to be read from the offsetdeserializer - The deserializer to be used to make a Cacheable from the data.IOExceptionpublic void write(ByteBuffer srcBuffer, long offset) throws IOException
write in interface IOEnginesrcBuffer - the given byte buffer from which bytes are to be readoffset - The offset in the ByteBufferArray of the first byte to be
writtenIOException - throws IOException if writing to the array throws exceptionpublic void write(ByteBuff srcBuffer, long offset) throws IOException
IOEnginewrite in interface IOEnginesrcBuffer - the given MultiBytebufffers from which bytes are to be readoffset - the offset in the IO engine where the first byte to be writtenIOExceptionpublic void sync()
Copyright © 2007–2020 The Apache Software Foundation. All rights reserved.