org.apache.hadoop.hbase.spark

HBaseContext

class HBaseContext extends Serializable with Logging

HBaseContext is a façade for HBase operations like bulk put, get, increment, delete, and scan

HBaseContext will take the responsibilities of disseminating the configuration information to the working and managing the life cycle of Connections.

Annotations
@Public()
Linear Supertypes
Logging, Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. HBaseContext
  2. Logging
  3. Serializable
  4. AnyRef
  5. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new HBaseContext(sc: SparkContext, config: Configuration, tmpHdfsConfgFile: String = null)

Type Members

  1. class WriterLength extends AnyRef

    This is a wrapper class around StoreFileWriter.

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any

  3. final def ##(): Int

    Definition Classes
    AnyRef → Any

  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any

  6. var appliedCredentials: Boolean

  7. def applyCreds[T](): Unit

  8. final def asInstanceOf[T0]: T0

    Definition Classes
    Any

  9. val broadcastedConf: Broadcast[SerializableWritable[Configuration]]

  10. def bulkDelete[T](rdd: RDD[T], tableName: TableName, f: (T) ⇒ Delete, batchSize: Integer): Unit

    A simple abstraction over the HBaseContext.

    A simple abstraction over the HBaseContext.foreachPartition method.

    It allow addition support for a user to take a RDD and generate delete and send them to HBase. The complexity of managing the Connection is removed from the developer

    rdd

    Original RDD with data to iterate over

    tableName

    The name of the table to delete from

    f

    Function to convert a value in the RDD to a HBase Deletes

    batchSize

    The number of delete to batch before sending to HBase

  11. def bulkGet[T, U](tableName: TableName, batchSize: Integer, rdd: RDD[T], makeGet: (T) ⇒ Get, convertResult: (Result) ⇒ U)(implicit arg0: ClassTag[U]): RDD[U]

    A simple abstraction over the HBaseContext.

    A simple abstraction over the HBaseContext.mapPartition method.

    It allow addition support for a user to take a RDD and generates a new RDD based on Gets and the results they bring back from HBase

    tableName

    The name of the table to get from

    rdd

    Original RDD with data to iterate over

    makeGet

    function to convert a value in the RDD to a HBase Get

    convertResult

    This will convert the HBase Result object to what ever the user wants to put in the resulting RDD return new RDD that is created by the Get to HBase

  12. def bulkLoad[T](rdd: RDD[T], tableName: TableName, flatMap: (T) ⇒ Iterator[(KeyFamilyQualifier, Array[Byte])], stagingDir: String, familyHFileWriteOptionsMap: Map[Array[Byte], FamilyHFileWriteOptions] = ..., compactionExclude: Boolean = false, maxSize: Long = HConstants.DEFAULT_MAX_FILE_SIZE): Unit

    Spark Implementation of HBase Bulk load for wide rows or when values are not already combined at the time of the map process

    Spark Implementation of HBase Bulk load for wide rows or when values are not already combined at the time of the map process

    This will take the content from an existing RDD then sort and shuffle it with respect to region splits. The result of that sort and shuffle will be written to HFiles.

    After this function is executed the user will have to call LoadIncrementalHFiles.doBulkLoad(...) to move the files into HBase

    Also note this version of bulk load is different from past versions in that it includes the qualifier as part of the sort process. The reason for this is to be able to support rows will very large number of columns.

    T

    The Type of values in the original RDD

    rdd

    The RDD we are bulk loading from

    tableName

    The HBase table we are loading into

    flatMap

    A flapMap function that will make every row in the RDD into N cells for the bulk load

    stagingDir

    The location on the FileSystem to bulk load into

    familyHFileWriteOptionsMap

    Options that will define how the HFile for a column family is written

    compactionExclude

    Compaction excluded for the HFiles

    maxSize

    Max size for the HFiles before they roll

  13. def bulkLoadThinRows[T](rdd: RDD[T], tableName: TableName, mapFunction: (T) ⇒ (ByteArrayWrapper, FamiliesQualifiersValues), stagingDir: String, familyHFileWriteOptionsMap: Map[Array[Byte], FamilyHFileWriteOptions] = ..., compactionExclude: Boolean = false, maxSize: Long = HConstants.DEFAULT_MAX_FILE_SIZE): Unit

    Spark Implementation of HBase Bulk load for short rows some where less then a 1000 columns.

    Spark Implementation of HBase Bulk load for short rows some where less then a 1000 columns. This bulk load should be faster for tables will thinner rows then the other spark implementation of bulk load that puts only one value into a record going into a shuffle

    This will take the content from an existing RDD then sort and shuffle it with respect to region splits. The result of that sort and shuffle will be written to HFiles.

    After this function is executed the user will have to call LoadIncrementalHFiles.doBulkLoad(...) to move the files into HBase

    In this implementation, only the rowKey is given to the shuffle as the key and all the columns are already linked to the RowKey before the shuffle stage. The sorting of the qualifier is done in memory out side of the shuffle stage

    Also make sure that incoming RDDs only have one record for every row key.

    T

    The Type of values in the original RDD

    rdd

    The RDD we are bulk loading from

    tableName

    The HBase table we are loading into

    mapFunction

    A function that will convert the RDD records to the key value format used for the shuffle to prep for writing to the bulk loaded HFiles

    stagingDir

    The location on the FileSystem to bulk load into

    familyHFileWriteOptionsMap

    Options that will define how the HFile for a column family is written

    compactionExclude

    Compaction excluded for the HFiles

    maxSize

    Max size for the HFiles before they roll

  14. def bulkPut[T](rdd: RDD[T], tableName: TableName, f: (T) ⇒ Put): Unit

    A simple abstraction over the HBaseContext.

    A simple abstraction over the HBaseContext.foreachPartition method.

    It allow addition support for a user to take RDD and generate puts and send them to HBase. The complexity of managing the Connection is removed from the developer

    rdd

    Original RDD with data to iterate over

    tableName

    The name of the table to put into

    f

    Function to convert a value in the RDD to a HBase Put

  15. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  16. val config: Configuration

  17. var credentials: Credentials

  18. val credentialsConf: Broadcast[SerializableWritable[Credentials]]

  19. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  20. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any

  21. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  22. def foreachPartition[T](dstream: DStream[T], f: (Iterator[T], Connection) ⇒ Unit): Unit

    A simple enrichment of the traditional Spark Streaming dStream foreach This function differs from the original in that it offers the developer access to a already connected Connection object

    A simple enrichment of the traditional Spark Streaming dStream foreach This function differs from the original in that it offers the developer access to a already connected Connection object

    Note: Do not close the Connection object. All Connection management is handled outside this method

    dstream

    Original DStream with data to iterate over

    f

    Function to be given a iterator to iterate through the DStream values and a Connection object to interact with HBase

  23. def foreachPartition[T](rdd: RDD[T], f: (Iterator[T], Connection) ⇒ Unit): Unit

    A simple enrichment of the traditional Spark RDD foreachPartition.

    A simple enrichment of the traditional Spark RDD foreachPartition. This function differs from the original in that it offers the developer access to a already connected Connection object

    Note: Do not close the Connection object. All Connection management is handled outside this method

    rdd

    Original RDD with data to iterate over

    f

    Function to be given a iterator to iterate through the RDD values and a Connection object to interact with HBase

  24. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any

  25. def hashCode(): Int

    Definition Classes
    AnyRef → Any

  26. def hbaseRDD(tableName: TableName, scans: Scan): RDD[(ImmutableBytesWritable, Result)]

    A overloaded version of HBaseContext hbaseRDD that defines the type of the resulting RDD

    A overloaded version of HBaseContext hbaseRDD that defines the type of the resulting RDD

    tableName

    the name of the table to scan

    scans

    the HBase scan object to use to read data from HBase

    returns

    New RDD with results from scan

  27. def hbaseRDD[U](tableName: TableName, scan: Scan, f: ((ImmutableBytesWritable, Result)) ⇒ U)(implicit arg0: ClassTag[U]): RDD[U]

    This function will use the native HBase TableInputFormat with the given scan object to generate a new RDD

    This function will use the native HBase TableInputFormat with the given scan object to generate a new RDD

    tableName

    the name of the table to scan

    scan

    the HBase scan object to use to read data from HBase

    f

    function to convert a Result object from HBase into what the user wants in the final generated RDD

    returns

    new RDD with results from scan

  28. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any

  29. def isTraceEnabled(): Boolean

    Attributes
    protected
    Definition Classes
    Logging

  30. val job: Job

  31. def log: Logger

    Attributes
    protected
    Definition Classes
    Logging

  32. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  33. def logDebug(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  34. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  35. def logError(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  36. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  37. def logInfo(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  38. def logName: String

    Attributes
    protected
    Definition Classes
    Logging

  39. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  40. def logTrace(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  41. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  42. def logWarning(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  43. def mapPartitions[T, R](rdd: RDD[T], mp: (Iterator[T], Connection) ⇒ Iterator[R])(implicit arg0: ClassTag[R]): RDD[R]

    A simple enrichment of the traditional Spark RDD mapPartition.

    A simple enrichment of the traditional Spark RDD mapPartition. This function differs from the original in that it offers the developer access to a already connected Connection object

    Note: Do not close the Connection object. All Connection management is handled outside this method

    rdd

    Original RDD with data to iterate over

    mp

    Function to be given a iterator to iterate through the RDD values and a Connection object to interact with HBase

    returns

    Returns a new RDD generated by the user definition function just like normal mapPartition

  44. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  45. final def notify(): Unit

    Definition Classes
    AnyRef

  46. final def notifyAll(): Unit

    Definition Classes
    AnyRef

  47. def streamBulkDelete[T](dstream: DStream[T], tableName: TableName, f: (T) ⇒ Delete, batchSize: Integer): Unit

    A simple abstraction over the HBaseContext.

    A simple abstraction over the HBaseContext.streamBulkMutation method.

    It allow addition support for a user to take a DStream and generate Delete and send them to HBase.

    The complexity of managing the Connection is removed from the developer

    dstream

    Original DStream with data to iterate over

    tableName

    The name of the table to delete from

    f

    function to convert a value in the DStream to a HBase Delete

    batchSize

    The number of deletes to batch before sending to HBase

  48. def streamBulkGet[T, U](tableName: TableName, batchSize: Integer, dStream: DStream[T], makeGet: (T) ⇒ Get, convertResult: (Result) ⇒ U)(implicit arg0: ClassTag[U]): DStream[U]

    A simple abstraction over the HBaseContext.

    A simple abstraction over the HBaseContext.streamMap method.

    It allow addition support for a user to take a DStream and generates a new DStream based on Gets and the results they bring back from HBase

    tableName

    The name of the table to get from

    batchSize

    The number of Gets to be sent in a single batch

    dStream

    Original DStream with data to iterate over

    makeGet

    Function to convert a value in the DStream to a HBase Get

    convertResult

    This will convert the HBase Result object to what ever the user wants to put in the resulting DStream

    returns

    A new DStream that is created by the Get to HBase

  49. def streamBulkPut[T](dstream: DStream[T], tableName: TableName, f: (T) ⇒ Put): Unit

    A simple abstraction over the HBaseContext.

    A simple abstraction over the HBaseContext.streamMapPartition method.

    It allow addition support for a user to take a DStream and generate puts and send them to HBase.

    The complexity of managing the Connection is removed from the developer

    dstream

    Original DStream with data to iterate over

    tableName

    The name of the table to put into

    f

    Function to convert a value in the DStream to a HBase Put

  50. def streamForeachPartition[T](dstream: DStream[T], f: (Iterator[T], Connection) ⇒ Unit): Unit

    A simple enrichment of the traditional Spark Streaming DStream foreachPartition.

    A simple enrichment of the traditional Spark Streaming DStream foreachPartition.

    This function differs from the original in that it offers the developer access to a already connected Connection object

    Note: Do not close the Connection object. All Connection management is handled outside this method

    Note: Make sure to partition correctly to avoid memory issue when getting data from HBase

    dstream

    Original DStream with data to iterate over

    f

    Function to be given a iterator to iterate through the DStream values and a Connection object to interact with HBase

    returns

    Returns a new DStream generated by the user definition function just like normal mapPartition

  51. def streamMapPartitions[T, U](dstream: DStream[T], f: (Iterator[T], Connection) ⇒ Iterator[U])(implicit arg0: ClassTag[U]): DStream[U]

    A simple enrichment of the traditional Spark Streaming DStream mapPartition.

    A simple enrichment of the traditional Spark Streaming DStream mapPartition.

    This function differs from the original in that it offers the developer access to a already connected Connection object

    Note: Do not close the Connection object. All Connection management is handled outside this method

    Note: Make sure to partition correctly to avoid memory issue when getting data from HBase

    dstream

    Original DStream with data to iterate over

    f

    Function to be given a iterator to iterate through the DStream values and a Connection object to interact with HBase

    returns

    Returns a new DStream generated by the user definition function just like normal mapPartition

  52. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef

  53. val tmpHdfsConfgFile: String

  54. var tmpHdfsConfiguration: Configuration

  55. def toString(): String

    Definition Classes
    AnyRef → Any

  56. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  57. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  58. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from Logging

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped