org.apache.hadoop.hbase.codec.prefixtree.encode.tokenize

Class TokenizerNode

java.lang.Object

org.apache.hadoop.hbase.codec.prefixtree.encode.tokenize.TokenizerNode

@InterfaceAudience.Private
public class TokenizerNode
extends Object

Individual node in a Trie structure. Each node is one of 3 types:

Branch: an internal trie node that may have a token and must have multiple children, but does not represent an actual input byte[], hence its numOccurrences is 0 Leaf: a node with no children and where numOccurrences is >= 1. It's token represents the last bytes in the input byte[]s. Nub: a combination of a branch and leaf. Its token represents the last bytes of input byte[]s and has numOccurrences >= 1, but it also has child nodes which represent input byte[]s that add bytes to this nodes input byte[].

Example inputs (numInputs=7): 0: AAA 1: AAA 2: AAB 3: AAB 4: AAB 5: AABQQ 6: AABQQ

Resulting TokenizerNodes: AA <- branch, numOccurrences=0, tokenStartOffset=0, token.length=2 A <- leaf, numOccurrences=2, tokenStartOffset=2, token.length=1 B <- nub, numOccurrences=3, tokenStartOffset=2, token.length=1 QQ <- leaf, numOccurrences=2, tokenStartOffset=3, token.length=2

numInputs == 7 == sum(numOccurrences) == 0 + 2 + 3 + 2