topic Re: How to set a global checkpoint for all notebooks? in Data Engineering https://community.databricks.com/t5/data-engineering/how-to-set-a-global-checkpoint-for-all-notebooks/m-p/20421#M13774 <P>@Alejandro Martinez​&nbsp;, For streaming jobs, there are, but others couldn't find them. Here are spark conf&nbsp;<A href="https://spark.apache.org/docs/latest/configuration.html" alt="https://spark.apache.org/docs/latest/configuration.html" target="_blank">Configuration - Spark 3.2.1 Documentation (apache.org)</A></P><PRE><CODE>spark.sql.streaming.checkpointLocation</CODE></PRE><P></P> Tue, 17 May 2022 09:32:42 GMT Hubert-Dudek 2022-05-17T09:32:42Z How to set a global checkpoint for all notebooks? https://community.databricks.com/t5/data-engineering/how-to-set-a-global-checkpoint-for-all-notebooks/m-p/20420#M13773 <P>I have several users doing data analysis on Databricks Spark notebooks, everything is smooth, now I want to make sure that the checkpointdir is configured on the cluster start, so every user doesn't had to set it on the Notebook (ending up in a lot of similar paths)</P><P>To do this:</P><PRE><CODE>SparkContext.checkpointFile(directory: String)</CODE></PRE><P>I had to have a context the ideal will be to set it here, on cluster configuration:</P><P><span class="lia-inline-image-display-wrapper" image-alt="image"><img src="https://community.databricks.com/t5/image/serverpage/image-id/1864i2722A16EDA8CADB9/image-size/large?v=v2&amp;px=999" role="button" title="image" alt="image" /></span>How can I achieve this?</P><P>Thanks!</P> Mon, 16 May 2022 18:49:16 GMT https://community.databricks.com/t5/data-engineering/how-to-set-a-global-checkpoint-for-all-notebooks/m-p/20420#M13773 alejandrofm 2022-05-16T18:49:16Z Re: How to set a global checkpoint for all notebooks? https://community.databricks.com/t5/data-engineering/how-to-set-a-global-checkpoint-for-all-notebooks/m-p/20421#M13774 <P>@Alejandro Martinez​&nbsp;, For streaming jobs, there are, but others couldn't find them. Here are spark conf&nbsp;<A href="https://spark.apache.org/docs/latest/configuration.html" alt="https://spark.apache.org/docs/latest/configuration.html" target="_blank">Configuration - Spark 3.2.1 Documentation (apache.org)</A></P><PRE><CODE>spark.sql.streaming.checkpointLocation</CODE></PRE><P></P> Tue, 17 May 2022 09:32:42 GMT https://community.databricks.com/t5/data-engineering/how-to-set-a-global-checkpoint-for-all-notebooks/m-p/20421#M13774 Hubert-Dudek 2022-05-17T09:32:42Z