How to set a global checkpoint for all notebooks?

alejandrofm
Valued Contributor

I have several users doing data analysis on Databricks Spark notebooks, everything is smooth, now I want to make sure that the checkpointdir is configured on the cluster start, so every user doesn't had to set it on the Notebook (ending up in a lot of similar paths)

To do this:

SparkContext.checkpointFile(directory: String)

I had to have a context the ideal will be to set it here, on cluster configuration:

imageHow can I achieve this?

Thanks!

Hubert-Dudek
Databricks MVP

@Alejandro Martinez​ , For streaming jobs, there are, but others couldn't find them. Here are spark conf Configuration - Spark 3.2.1 Documentation (apache.org)

spark.sql.streaming.checkpointLocation


My blog: https://databrickster.medium.com/

View solution in original post