Package org.apache.hadoop.hbase

Class TestPartialResultsFromClientSide

java.lang.Object
org.apache.hadoop.hbase.TestPartialResultsFromClientSide
@Tag("org.apache.hadoop.hbase.testclassification.LargeTests") public class TestPartialResultsFromClientSide extends Object
These tests are focused on testing how partial results appear to a client. Partial results are Results that contain only a portion of a row's complete list of cells. Partial results are formed when the server breaches its maximum result size when trying to service a client's RPC request. It is the responsibility of the scanner on the client side to recognize when partial results have been returned and to take action to form the complete results.

Unless the flag Scan.setAllowPartialResults(boolean) has been set to true, the caller of ResultScanner.next() should never see partial results.

  • Field Details

  • Constructor Details

  • Method Details

    • setUpBeforeClass

      @BeforeAll public static void setUpBeforeClass() throws Exception
      Throws:
      Exception

    • createTestTable

      static org.apache.hadoop.hbase.client.Table createTestTable(org.apache.hadoop.hbase.TableName name, byte[][] rows, byte[][] families, byte[][] qualifiers, byte[] cellValue) throws IOException
      Throws:
      IOException

    • tearDownAfterClass

      @AfterAll public static void tearDownAfterClass() throws Exception
      Throws:
      Exception

    • testExpectedValuesOfPartialResults

      @Test public void testExpectedValuesOfPartialResults() throws Exception
      Ensure that the expected key values appear in a result returned from a scanner that is combining partial results into complete results
      Throws:
      Exception

    • testExpectedValuesOfPartialResults

      public void testExpectedValuesOfPartialResults(boolean reversed) throws Exception
      Throws:
      Exception

    • testAllowPartialResults

      @Test public void testAllowPartialResults() throws Exception
      Ensure that we only see Results marked as partial when the allowPartial flag is set
      Throws:
      Exception

    • testEquivalenceOfScanResults

      @Test public void testEquivalenceOfScanResults() throws Exception
      Ensure that the results returned from a scanner that retrieves all results in a single RPC call matches the results that are returned from a scanner that must incrementally combine partial results into complete results. A variety of scan configurations can be tested
      Throws:
      Exception

    • testEquivalenceOfScanResults

      public void testEquivalenceOfScanResults(org.apache.hadoop.hbase.client.Table table, org.apache.hadoop.hbase.client.Scan scan1, org.apache.hadoop.hbase.client.Scan scan2) throws Exception
      Throws:
      Exception

    • testOrderingOfCellsInPartialResults

      @Test public void testOrderingOfCellsInPartialResults() throws Exception
      Order of cells in partial results matches the ordering of cells from complete results
      Throws:
      Exception

    • testOrderingOfCellsInPartialResults

      public void testOrderingOfCellsInPartialResults(org.apache.hadoop.hbase.client.Scan basePartialScan) throws Exception
      Throws:
      Exception

    • testExpectedNumberOfCellsPerPartialResult

      @Test public void testExpectedNumberOfCellsPerPartialResult() throws Exception
      Setting the max result size allows us to control how many cells we expect to see on each call to next on the scanner. Test a variety of different sizes for correctness
      Throws:
      Exception

    • testExpectedNumberOfCellsPerPartialResult

      public void testExpectedNumberOfCellsPerPartialResult(org.apache.hadoop.hbase.client.Scan baseScan) throws Exception
      Throws:
      Exception

    • testExpectedNumberOfCellsPerPartialResult

      public void testExpectedNumberOfCellsPerPartialResult(org.apache.hadoop.hbase.client.Scan baseScan, int expectedNumberOfCells) throws Exception
      Throws:
      Exception

    • getCellHeapSize

      private long getCellHeapSize() throws Exception
      Returns:
      The approximate heap size of a cell in the test table. All cells should have approximately the same heap size, so the value is cached to avoid repeating the calculation
      Throws:
      Exception

    • getResultSizeForNumberOfCells

      private long getResultSizeForNumberOfCells(int numberOfCells) throws Exception
      Returns:
      the result size that should be used in Scan.setMaxResultSize(long) if you want the server to return exactly numberOfCells cells
      Throws:
      Exception

    • testPartialResultsAndBatch

      @Test public void testPartialResultsAndBatch() throws Exception
      Test various combinations of batching and partial results for correctness
      Throws:
      Exception

    • testPartialResultsAndBatch

      public void testPartialResultsAndBatch(int batch, int cellsPerPartialResult) throws Exception
      Throws:
      Exception

    • testPartialResultsReassembly

      @Test public void testPartialResultsReassembly() throws Exception
      Test the method Result.createCompleteResult(Iterable)
      Throws:
      Exception

    • testPartialResultsReassembly

      public void testPartialResultsReassembly(org.apache.hadoop.hbase.client.Scan scanBase) throws Exception
      Throws:
      Exception

    • testExceptionThrownOnMismatchedPartialResults

      @Test public void testExceptionThrownOnMismatchedPartialResults() throws IOException
      When reconstructing the complete result from its partials we ensure that the row of each partial result is the same. If one of the rows differs, an exception is thrown.
      Throws:
      IOException

    • testNoPartialResultsWhenRowFilterPresent

      @Test public void testNoPartialResultsWhenRowFilterPresent() throws Exception
      When a scan has a filter where Filter.hasFilterRow() is true, the scanner should not return partial results. The scanner cannot return partial results because the entire row needs to be read for the include/exclude decision to be made
      Throws:
      Exception

    • testPartialResultsAndCaching

      @Test public void testPartialResultsAndCaching() throws Exception
      Examine the interaction between the maxResultSize and caching. If the caching limit is reached before the maxResultSize limit, we should not see partial results. On the other hand, if the maxResultSize limit is reached before the caching limit, it is likely that partial results will be seen.
      Throws:
      Exception

    • testPartialResultsAndCaching

      public void testPartialResultsAndCaching(int resultSizeRowLimit, int cachingRowLimit) throws Exception
      Parameters:
      resultSizeRowLimit - The row limit that will be enforced through maxResultSize
      cachingRowLimit - The row limit that will be enforced through caching
      Throws:
      Exception

    • createPuts

      static ArrayList<org.apache.hadoop.hbase.client.Put> createPuts(byte[][] rows, byte[][] families, byte[][] qualifiers, byte[] value) throws IOException
      Make puts to put the input value into each combination of row, family, and qualifier
      Parameters:
      rows - the rows to use
      families - the families to use
      qualifiers - the qualifiers to use
      value - the values to use
      Returns:
      the dot product of the given rows, families, qualifiers, and values
      Throws:
      IOException - if there is a problem creating one of the Put objects

    • createKeyValuesForRow

      static ArrayList<org.apache.hadoop.hbase.Cell> createKeyValuesForRow(byte[] row, byte[][] families, byte[][] qualifiers, byte[] value)
      Make key values to represent each possible combination of family and qualifier in the specified row.
      Parameters:
      row - the row to use
      families - the families to use
      qualifiers - the qualifiers to use
      value - the values to use
      Returns:
      the dot product of the given families, qualifiers, and values for a given row

    • verifyResult

      static void verifyResult(org.apache.hadoop.hbase.client.Result result, List<org.apache.hadoop.hbase.Cell> expKvList, String msg)
      Verifies that result contains all the key values within expKvList. Fails the test otherwise

    • compareResults

      static void compareResults(org.apache.hadoop.hbase.client.Result r1, org.apache.hadoop.hbase.client.Result r2, String message)
      Compares two results and fails the test if the results are different

    • testReadPointAndPartialResults

      @Test public void testReadPointAndPartialResults(org.junit.jupiter.api.TestInfo testInfo) throws Exception
      Throws:
      Exception

    • countCellsFromScanner

      private int countCellsFromScanner(org.apache.hadoop.hbase.client.ResultScanner scanner) throws Exception
      Exhausts the scanner by calling next repetitively. Once completely exhausted, close scanner and return total cell count
      Parameters:
      scanner - the scanner to exhaust
      Returns:
      the number of cells counted
      Throws:
      Exception - if there is a problem retrieving cells from the scanner

    • testPartialResultsWithColumnFilter

      @Test public void testPartialResultsWithColumnFilter() throws Exception
      Test partial Result re-assembly in the presence of different filters. The Results from the partial scanner should match the Results returned from a scanner that receives all of the results in one RPC to the server. The partial scanner is tested with a variety of different result sizes (all of which are less than the size necessary to fetch an entire row)
      Throws:
      Exception

    • testPartialResultsWithColumnFilter

      public void testPartialResultsWithColumnFilter(org.apache.hadoop.hbase.filter.Filter filter) throws Exception
      Throws:
      Exception

    • moveRegion

      private void moveRegion(org.apache.hadoop.hbase.client.Table table, int index) throws IOException
      Throws:
      IOException

    • assertCell

      private void assertCell(org.apache.hadoop.hbase.Cell cell, byte[] row, byte[] cf, byte[] cq)

    • testPartialResultWhenRegionMove

      @Test public void testPartialResultWhenRegionMove(org.junit.jupiter.api.TestInfo testInfo) throws IOException
      Throws:
      IOException

    • testReversedPartialResultWhenRegionMove

      @Test public void testReversedPartialResultWhenRegionMove(org.junit.jupiter.api.TestInfo testInfo) throws IOException
      Throws:
      IOException

    • testCompleteResultWhenRegionMove

      @Test public void testCompleteResultWhenRegionMove(org.junit.jupiter.api.TestInfo testInfo) throws IOException
      Throws:
      IOException

    • testReversedCompleteResultWhenRegionMove

      @Test public void testReversedCompleteResultWhenRegionMove(org.junit.jupiter.api.TestInfo testInfo) throws IOException
      Throws:
      IOException

    • testBatchingResultWhenRegionMove

      @Test public void testBatchingResultWhenRegionMove(org.junit.jupiter.api.TestInfo testInfo) throws IOException
      Throws:
      IOException

    • testDontThrowUnknowScannerExceptionToClient

      @Test public void testDontThrowUnknowScannerExceptionToClient(org.junit.jupiter.api.TestInfo testInfo) throws Exception
      Throws:
      Exception

    • testMayHaveMoreCellsInRowReturnsTrueAndSetBatch

      @Test public void testMayHaveMoreCellsInRowReturnsTrueAndSetBatch(org.junit.jupiter.api.TestInfo testInfo) throws IOException
      Throws:
      IOException