topic Re: Runtime SQL Configuration - how to make it simple in Data Engineering

Runtime SQL Configuration - how to make it simple

Leszek — Wed, 17 Nov 2021 15:00:29 GMT

Hi, I'm running couple of Notebooks in my pipeline and I would like to set fixed value of 'spark.sql.shuffle.partitions' - same value for every notebook. Should I do that by adding spark.conf.set.. code in each Notebook (Runtime SQL configurations are per-session) or is there any other, easier way to set this?

Re: Runtime SQL Configuration - how to make it simple

Hubert-Dudek — Wed, 17 Nov 2021 15:30:08 GMT

Easiest is to use spark config in advanced options in cluster settings:

more info here https://docs.databricks.com/clusters/configure.html#spark-configuration

Re: Runtime SQL Configuration - how to make it simple

Prabakar — Wed, 17 Nov 2021 16:01:17 GMT

Setting it either in the notebook or the cluster, both should work. But the better option would be to go with the cluster's spark config.

Re: Runtime SQL Configuration - how to make it simple

jose_gonzalez — Wed, 17 Nov 2021 18:21:02 GMT

Hi @Leszek ,

Like @Hubert Dudek mentioned, I will recommend to add your setting to the cluster Spark configurations.

The difference between notebook vs cluster, is that:

In the Notebook, the Spark configurations will only apply to the notebook Spark context itself. The Spark configurations only will apply to the notebook
In the cluster, the Spark setting will be global and it will be apply to all the notebooks that are attach to the cluster.

Re: Runtime SQL Configuration - how to make it simple

Leszek — Thu, 18 Nov 2021 07:41:57 GMT

Hi, Thank you all for the tips. I tried before to set this option in Spark Config but didn't work for some reason. Today I tried again and it's working :).

Re: Runtime SQL Configuration - how to make it simple

Hubert-Dudek — Thu, 18 Nov 2021 12:00:07 GMT

Great that it is working. Any chance to be selected as best answer? 🙂