jose_gonzalez
Databricks Employee
Databricks Employee

Hi @Leszek​ ,

Like @Hubert Dudek​ mentioned, I will recommend to add your setting to the cluster Spark configurations.

The difference between notebook vs cluster, is that:

  • In the Notebook, the Spark configurations will only apply to the notebook Spark context itself. The Spark configurations only will apply to the notebook
  • In the cluster, the Spark setting will be global and it will be apply to all the notebooks that are attach to the cluster.