topic Re: Azure Databricks: How to add Spark configuration in Databricks cluster? in Data Engineering

Azure Databricks: How to add Spark configuration in Databricks cluster?

User16790091296 — Thu, 24 Jun 2021 15:27:21 GMT

I am using a Spark Databricks cluster and want to add a customized Spark configuration.

There is a Databricks documentation on this but I am not getting any clue how and what changes I should make. Can someone pls share the example to configure the Databricks cluster.

Is there any way to see the default configuration for Spark in the Databricks cluster.

Re: Azure Databricks: How to add Spark configuration in Databricks cluster?

brickster_2018 — Thu, 24 Jun 2021 23:59:49 GMT

You can set the configurations on the Databricks cluster UI

https://docs.databricks.com/clusters/configure.html#spark-configuration

To see the default configuration, run the below code in a notebook:

%sql
set;