โ05-03-2015 12:28 PM
โ05-03-2015 12:35 PM
From the Clusters tab, select a cluster and view the Spark UI.
The Environment tab shows the current Spark configuration settings.
Here is an exhaustive list of the Spark Config params: https://spark.apache.org/docs/latest/configuration.html
The
SparkContext
is provided for you within the notebook UI, therefore you cannot change these values within your notebook code. Once SparkConf
is passed to the SparkContext
constructor, the values are cloned and cannot be changed. This is a Spark limitation.One thing to note is that Databricks has already tuned Spark for the most common workloads running on the specific EC2 instance types used within Databricks Cloud.
In other words, you shouldn't have to changes these default values except in extreme cases. To change these defaults, please contact Databricks Cloud support.
If you're working with the SqlContext or HiveContext, you can manually set configuration properties using HiveQL's
SET key=value
command with spark.sql.*
properties from this list, for example: https://spark.apache.org/docs/latest/sql-programming-guide.html#configuration.However, overriding these configuration values may cause problems for other users of the cluster.
โ06-15-2016 11:19 AM
How would we handle the config for elasticsearch?
https://www.elastic.co/guide/en/elasticsearch/hadoop/current/spark.html#spark-native-cfg
โ03-02-2017 08:21 AM
hi, may I know how did you handle the config for elasticsearch? I also have to stream data to elasticsearch.
โ03-04-2017 07:29 AM
There is a 'spark' tab in the cluster creation page, you can add the configs there before starting the cluster.
โ01-13-2019 05:32 PM
This is an old post, however, is this still accurate for the latest version of Databricks in 2019? If so, how to approach the following?
1. Connect to many MongoDBs.2. Connect to MongoDB when connection string information is dynamic (i.e. stored in spark table).Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโt want to miss the chance to attend and share knowledge.
If there isnโt a group near you, start one and help create a community that brings people together.
Request a New Group