How to set a global checkpoint for all notebooks?

[alejandrofm]

I have several users doing data analysis on Databricks Spark notebooks, everything is smooth, now I want to make sure that the checkpointdir is configured on the cluster start, so every user doesn't had to set it on the Notebook (ending up in a lot of similar paths)

To do this:

SparkContext.checkpointFile(directory: String)

I had to have a context the ideal will be to set it here, on cluster configuration:

How can I achieve this?

Thanks!

[Hubert-Dudek · Accepted solution]

@Alejandro Martinez​ , For streaming jobs, there are, but others couldn't find them. Here are spark conf Configuration - Spark 3.2.1 Documentation (apache.org)

spark.sql.streaming.checkpointLocation

[Kaniz]

Hi @Alejandro Martinez​ , Just a friendly follow-up. Do you still need help, or @Hubert Dudek (Customer)​ 's response help you to find the solution? Please let us know.