org.apache.hadoop.hbase.io.hfile.bucket

Class BucketAllocator.IndexStatistics

java.lang.Object

org.apache.hadoop.hbase.io.hfile.bucket.BucketAllocator.IndexStatistics

Enclosing class:

BucketAllocator

static class BucketAllocator.IndexStatistics
extends Object

Statistics to give a glimpse into the distribution of BucketCache objects. Each configured bucket size, denoted by BucketAllocator.BucketSizeInfo, gets an IndexStatistic. A BucketSizeInfo allocates blocks of a configured size from claimed buckets. If you have a bucket size of 512k, the corresponding BucketSizeInfo will always allocate chunks of 512k at a time regardless of actual request.

Over time, as a BucketSizeInfo gets more allocations, it will claim more buckets from the total pool of completelyFreeBuckets. As blocks are freed from a BucketSizeInfo, those buckets may be returned to the completelyFreeBuckets pool.

The IndexStatistics help visualize how these buckets are currently distributed, through counts of items, bytes, and fullBuckets. Additionally, mismatches between block sizes and bucket sizes can manifest in inefficient cache usage. These typically manifest in three ways:

1. Allocation failures, because block size is larger than max bucket size. These show up in logs and can be alleviated by adding larger bucket sizes if appropriate.
2. Memory fragmentation, because blocks are typically smaller than the bucket size. See fragmentationBytes() for details.
3. Memory waste, because a bucket's itemSize is not a perfect divisor of bucketCapacity. see wastedBytes() for details.

Field Summary

Fields

Modifier and Type	Field and Description
private int	completelyFreeBuckets
private long	fragmentationBytes
private long	freeCount
private int	fullBuckets
private long	itemSize
private long	totalCount
private long	usedCount
private long	wastedBytes

Constructor Summary

Constructors

Constructor and Description
IndexStatistics()
IndexStatistics(long free, long used, long itemSize, int fullBuckets, int completelyFreeBuckets, long wastedBytes, long fragmentationBytes)

Method Summary

Modifier and Type	Method and Description
int	completelyFreeBuckets() How many buckets are currently claimed by this bucket size but as yet totally unused.
long	fragmentationBytes() Every time you allocate blocks in these buckets where the block size is less than the bucket size, fragmentation increases by that difference.
long	freeBytes() How many more bytes can be allocated from the currently claimed blocks of this bucket size
long	freeCount() How many more items can be allocated from the currently claimed blocks of this bucket size
int	fullBuckets() How many buckets have been completely filled by blocks for this bucket size.
long	itemSize() This bucket size can only allocate items of this size, even if the requested allocation size is smaller.
void	setTo(long free, long used, long itemSize, int fullBuckets, int completelyFreeBuckets, long wastedBytes, long fragmentationBytes)
long	totalBytes() Combined totalCount() * itemSize()
long	totalCount() Combined freeCount() + usedCount()
long	usedBytes() How many bytes are currently taking up space in this bucket size's buckets Note: If your items are less than the bucket size of this bucket, the actual used bytes by items will be lower than this value.
long	usedCount() How many items are currently taking up space in this bucket size's buckets
long	wastedBytes() If BucketAllocator.bucketCapacity is not perfectly divisible by this itemSize(), the remainder will be unusable by in buckets of this size.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

freeCount

private long freeCount

usedCount

private long usedCount

itemSize

private long itemSize

totalCount

private long totalCount

wastedBytes

private long wastedBytes

fragmentationBytes

private long fragmentationBytes

fullBuckets

private int fullBuckets

completelyFreeBuckets

private int completelyFreeBuckets

Constructor Detail

IndexStatistics

public IndexStatistics(long free,
                       long used,
                       long itemSize,
                       int fullBuckets,
                       int completelyFreeBuckets,
                       long wastedBytes,
                       long fragmentationBytes)

IndexStatistics

public IndexStatistics()

Method Detail

freeCount

public long freeCount()

How many more items can be allocated from the currently claimed blocks of this bucket size

usedCount

public long usedCount()

How many items are currently taking up space in this bucket size's buckets

totalCount

public long totalCount()

Combined freeCount() + usedCount()

freeBytes

public long freeBytes()

How many more bytes can be allocated from the currently claimed blocks of this bucket size

usedBytes

public long usedBytes()

How many bytes are currently taking up space in this bucket size's buckets Note: If your items are less than the bucket size of this bucket, the actual used bytes by items will be lower than this value. But since a bucket size can only allocate items of a single size, this value is the true number of used bytes. The difference will be counted in fragmentationBytes().

totalBytes

public long totalBytes()

Combined totalCount() * itemSize()

itemSize

public long itemSize()

This bucket size can only allocate items of this size, even if the requested allocation size is smaller. The rest goes towards fragmentationBytes().

fullBuckets

public int fullBuckets()

How many buckets have been completely filled by blocks for this bucket size. These buckets can't accept any more blocks unless some existing are freed.

completelyFreeBuckets

public int completelyFreeBuckets()

How many buckets are currently claimed by this bucket size but as yet totally unused. These buckets are available for reallocation to other bucket sizes if those fill up.

wastedBytes

public long wastedBytes()

If BucketAllocator.bucketCapacity is not perfectly divisible by this itemSize(), the remainder will be unusable by in buckets of this size. A high value here may be optimized by trying to choose bucket sizes which can better divide BucketAllocator.bucketCapacity.

fragmentationBytes

public long fragmentationBytes()

Every time you allocate blocks in these buckets where the block size is less than the bucket size, fragmentation increases by that difference. You can reduce fragmentation by lowering the bucket size so that it is closer to the typical block size. This may have the consequence of bumping some blocks to the next larger bucket size, so experimentation may be needed.

setTo

public void setTo(long free,
                  long used,
                  long itemSize,
                  int fullBuckets,
                  int completelyFreeBuckets,
                  long wastedBytes,
                  long fragmentationBytes)