Packages

package util

Spark utilities.

Source
package.scala
Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. util
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Package Members

  1. package random

    Utilities for random number generation.

  2. package sketch

Type Members

  1. abstract class AccumulatorV2[IN, OUT] extends Serializable

    The base class for accumulators, that can accumulate inputs of type IN, and produce output of type OUT.

    The base class for accumulators, that can accumulate inputs of type IN, and produce output of type OUT.

    OUT should be a type that can be read atomically (e.g., Int, Long), or thread-safely (e.g., synchronized collections) because it will be read from other threads.

  2. class ChildFirstURLClassLoader extends MutableURLClassLoader

    A mutable class loader that gives preference to its own URLs over the parent class loader when loading classes and resources.

  3. class CollectionAccumulator[T] extends AccumulatorV2[T, List[T]]

    An accumulator for collecting a list of elements.

    An accumulator for collecting a list of elements.

    Since

    2.0.0

  4. class DoubleAccumulator extends AccumulatorV2[Double, Double]

    An accumulator for computing sum, count, and averages for double precision floating numbers.

    An accumulator for computing sum, count, and averages for double precision floating numbers.

    Since

    2.0.0

  5. class EnumUtil extends AnyRef
    Annotations
    @Private()
  6. trait LexicalThreadLocal[T] extends AnyRef

    Helper trait for defining thread locals with lexical scoping.

    Helper trait for defining thread locals with lexical scoping. With this helper, the thread local is private and can only be set by the Handle. The Handle only exposes the thread local value to functions passed into its runWith method. This pattern allows for the lifetime of the thread local value to be strictly controlled.

    Rather than calling tl.set(...) and tl.remove() you would get a handle and execute your code in handle.runWith { ... }.

    Example:

    object Credentials extends LexicalThreadLocal[Int] {
  def create(creds: Map[String, String]) = new Handle(Some(creds))
}
...
val handle = Credentials.create(Map("key" -> "value"))
assert(Credentials.get() == None)
handle.runWith {
  assert(Credentials.get() == Some(Map("key" -> "value")))
}
  7. class LongAccumulator extends AccumulatorV2[Long, Long]

    An accumulator for computing sum, count, and average of 64-bit integers.

    An accumulator for computing sum, count, and average of 64-bit integers.

    Since

    2.0.0

  8. case class MutablePair[T1, T2](_1: T1, _2: T2) extends Product2[T1, T2] with Product with Serializable

    :: DeveloperApi :: A tuple of 2 elements.

    :: DeveloperApi :: A tuple of 2 elements. This can be used as an alternative to Scala's Tuple2 when we want to minimize object allocation.

    _1

    Element 1 of this MutablePair

    _2

    Element 2 of this MutablePair

    Annotations
    @DeveloperApi()
  9. class MutableURLClassLoader extends URLClassLoader

    URL class loader that exposes the addURL method in URLClassLoader.

  10. class ParentClassLoader extends ClassLoader

    A class loader which makes some protected methods in ClassLoader accessible.

  11. class SerializableConfiguration extends Serializable

    Hadoop configuration but serializable.

    Hadoop configuration but serializable. Use value to access the Hadoop configuration.

    Annotations
    @DeveloperApi() @Unstable()
  12. class StatCounter extends Serializable

    A class for tracking the statistics of a set of numbers (count, mean and variance) in a numerically robust way.

    A class for tracking the statistics of a set of numbers (count, mean and variance) in a numerically robust way. Includes support for merging two StatCounters. Based on Welford and Chan's algorithms for running variance.

  13. trait TaskCompletionListener extends EventListener

    :: DeveloperApi ::

    :: DeveloperApi ::

    Listener providing a callback function to invoke when a task's execution completes.

    Annotations
    @DeveloperApi()
  14. trait TaskFailureListener extends EventListener

    :: DeveloperApi ::

    :: DeveloperApi ::

    Listener providing a callback function to invoke when a task's execution encounters an error. Operations defined here must be idempotent, as onTaskFailure can be called multiple times.

    Annotations
    @DeveloperApi()

Value Members

  1. object LogUtils

    :: : DeveloperApi :: Utils for querying Spark logs with Spark SQL.

    :: : DeveloperApi :: Utils for querying Spark logs with Spark SQL.

    Annotations
    @DeveloperApi()
    Since

    4.0.0

  2. object SizeEstimator extends Logging

    :: DeveloperApi :: Estimates the sizes of Java objects (number of bytes of memory they occupy), for use in memory-aware caches.

    :: DeveloperApi :: Estimates the sizes of Java objects (number of bytes of memory they occupy), for use in memory-aware caches.

    Based on the following JavaWorld article: https://www.infoworld.com/article/2077408/sizeof-for-java.html

    Annotations
    @DeveloperApi()
  3. object SparkEnvUtils extends SparkEnvUtils
  4. object StatCounter extends Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped