Class SparkSession.Builder

Method Details

• appName

  public SparkSession.Builder appName(String name)

  Sets a name for the application, which will be shown in the Spark web UI. If no application name is set, a randomly generated name will be used.

  Parameters:

  name - (undocumented)

  Returns:

  (undocumented)

  Inheritdoc:

• classic

  public SparkSession.Builder classic()

  Make the builder create a Classic SparkSession.

  Returns:

  (undocumented)

• config

  public SparkSession.Builder config(String key, String value)

  Sets a config option. Options set using this method are automatically propagated to both SparkConf and SparkSession's own configuration.

  Parameters:

  key - (undocumented)

  value - (undocumented)

  Returns:

  (undocumented)

  Inheritdoc:

• config

  public SparkSession.Builder config(String key, long value)

  Sets a config option. Options set using this method are automatically propagated to both SparkConf and SparkSession's own configuration.

  Parameters:

  key - (undocumented)

  value - (undocumented)

  Returns:

  (undocumented)

  Inheritdoc:

• config

  public SparkSession.Builder config(String key, double value)

  Sets a config option. Options set using this method are automatically propagated to both SparkConf and SparkSession's own configuration.

  Parameters:

  key - (undocumented)

  value - (undocumented)

  Returns:

  (undocumented)

  Inheritdoc:

• config

  public SparkSession.Builder config(String key, boolean value)

  Sets a config option. Options set using this method are automatically propagated to both SparkConf and SparkSession's own configuration.

  Parameters:

  key - (undocumented)

  value - (undocumented)

  Returns:

  (undocumented)

  Inheritdoc:

• config

  public SparkSession.Builder config(scala.collection.immutable.Map<String,Object> map)

  Sets a config option. Options set using this method are automatically propagated to both SparkConf and SparkSession's own configuration.

  Parameters:

  map - (undocumented)

  Returns:

  (undocumented)

  Inheritdoc:

• config

  public SparkSession.Builder config(Map<String,Object> map)

  Sets a config option. Options set using this method are automatically propagated to both SparkConf and SparkSession's own configuration.

  Parameters:

  map - (undocumented)

  Returns:

  (undocumented)

  Inheritdoc:

• config

  public SparkSession.Builder config(SparkConf conf)

  Sets a list of config options based on the given SparkConf.

  Parameters:

  conf - (undocumented)

  Returns:

  (undocumented)

  Inheritdoc:

• connect

  public SparkSession.Builder connect()

  Make the builder create a Connect SparkSession.

  Returns:

  (undocumented)

• create

  public SparkSession create()

  Create a new SparkSession.

  This will always return a newly created session.

  This method will update the default and/or active session if they are not set.

  Returns:

  (undocumented)

  Inheritdoc:

• enableHiveSupport

  public SparkSession.Builder enableHiveSupport()

  Enables Hive support, including connectivity to a persistent Hive metastore, support for Hive serdes, and Hive user-defined functions.

  Returns:

  (undocumented)

  Inheritdoc:

• getOrCreate

  public SparkSession getOrCreate()

  Gets an existing SparkSession or, if there is no existing one, creates a new one based on the options set in this builder.

  This method first checks whether there is a valid thread-local SparkSession, and if yes, return that one. It then checks whether there is a valid global default SparkSession, and if yes, return that one. If no valid global default SparkSession exists, the method creates a new SparkSession and assigns the newly created SparkSession as the global default.

  In case an existing SparkSession is returned, the non-static config options specified in this builder will be applied to the existing SparkSession.

  Returns:

  (undocumented)

  Inheritdoc:

• master

  public SparkSession.Builder master(String master)

  Sets the Spark master URL to connect to, such as "local" to run locally, "local[4]" to run locally with 4 cores, or "spark://master:7077" to run on a Spark standalone cluster.

  Parameters:

  master - (undocumented)

  Returns:

  (undocumented)

  Inheritdoc:

• remote

  public SparkSession.Builder remote(String connectionString)

  Sets the Spark Connect remote URL.

  Parameters:

  connectionString - (undocumented)

  Returns:

  (undocumented)

  Inheritdoc:

• withExtensions

  public SparkSession.Builder withExtensions(scala.Function1<SparkSessionExtensions,scala.runtime.BoxedUnit> f)

  Inject extensions into the SparkSession. This allows a user to add Analyzer rules, Optimizer rules, Planning Strategies or a customized parser.

  Parameters:

  f - (undocumented)

  Returns:

  (undocumented)

  Inheritdoc:

• MASTER_KEY

  public static String MASTER_KEY()

• APP_NAME_KEY

  public static String APP_NAME_KEY()

• CATALOG_IMPL_KEY

  public static String CATALOG_IMPL_KEY()

• CONNECT_REMOTE_KEY

  public static String CONNECT_REMOTE_KEY()

• API_MODE_KEY

  public static String API_MODE_KEY()

• API_MODE_CLASSIC

  public static String API_MODE_CLASSIC()

• API_MODE_CONNECT

  public static String API_MODE_CONNECT()