topic Re: How can I view and change the SparkConf settings if the SparkContext (sc) is already provided for me? in Data Engineering

How can I view and change the SparkConf settings if the SparkContext (sc) is already provided for me?

cfregly — Sun, 03 May 2015 19:28:53 GMT

Re: How can I view and change the SparkConf settings if the SparkContext (sc) is already provided for me?

cfregly — Sun, 03 May 2015 19:35:13 GMT

From the Clusters tab, select a cluster and view the Spark UI.

The Environment tab shows the current Spark configuration settings.

Here is an exhaustive list of the Spark Config params: https://spark.apache.org/docs/latest/configuration.html

The

SparkContext

is provided for you within the notebook UI, therefore you cannot change these values within your notebook code. Once

SparkConf

is passed to the

SparkContext

constructor, the values are cloned and cannot be changed. This is a Spark limitation.

One thing to note is that Databricks has already tuned Spark for the most common workloads running on the specific EC2 instance types used within Databricks Cloud.

In other words, you shouldn't have to changes these default values except in extreme cases. To change these defaults, please contact Databricks Cloud support.

If you're working with the SqlContext or HiveContext, you can manually set configuration properties using HiveQL's

SET key=value

command with

spark.sql.*

properties from this list, for example: https://spark.apache.org/docs/latest/sql-programming-guide.html#configuration.

However, overriding these configuration values may cause problems for other users of the cluster.

Re: How can I view and change the SparkConf settings if the SparkContext (sc) is already provided for me?

JonathanSpooner — Wed, 15 Jun 2016 18:19:37 GMT

How would we handle the config for elasticsearch?

https://www.elastic.co/guide/en/elasticsearch/hadoop/current/spark.html#spark-native-cfg

Re: How can I view and change the SparkConf settings if the SparkContext (sc) is already provided for me?

Shanmugavel — Thu, 02 Mar 2017 16:21:16 GMT

hi, may I know how did you handle the config for elasticsearch? I also have to stream data to elasticsearch.

Re: How can I view and change the SparkConf settings if the SparkContext (sc) is already provided for me?

User16789201666 — Sat, 04 Mar 2017 15:29:45 GMT

There is a 'spark' tab in the cluster creation page, you can add the configs there before starting the cluster.

Re: How can I view and change the SparkConf settings if the SparkContext (sc) is already provided for me?

MatthewValenti — Mon, 14 Jan 2019 01:32:03 GMT

This is an old post, however, is this still accurate for the latest version of Databricks in 2019? If so, how to approach the following?

1. Connect to many MongoDBs.

2. Connect to MongoDB when connection string information is dynamic (i.e. stored in spark table).