https://community.databricks.com/t5/data-engineering/spark-sparkcontext-setcheckpointdir-external-azure-storage/m-p/5311#M1762 <P>Is it possible to direct spark.sparkContext.setCheckpointDir to an external Azure Storage Container location (instead of DBFS) &amp; if so how, there's very little documentation on that.</P> Tue, 25 Apr 2023 08:40:42 GMT sintsan 2023-04-25T08:40:42Z https://community.databricks.com/t5/data-engineering/spark-sparkcontext-setcheckpointdir-external-azure-storage/m-p/5311#M1762 <P>Is it possible to direct spark.sparkContext.setCheckpointDir to an external Azure Storage Container location (instead of DBFS) &amp; if so how, there's very little documentation on that.</P> Tue, 25 Apr 2023 08:40:42 GMT https://community.databricks.com/t5/data-engineering/spark-sparkcontext-setcheckpointdir-external-azure-storage/m-p/5311#M1762 sintsan 2023-04-25T08:40:42Z https://community.databricks.com/t5/data-engineering/spark-sparkcontext-setcheckpointdir-external-azure-storage/m-p/5312#M1763 <P>yes,</P><P>the directory must be an HDFS path if running on a cluster.</P><P>All you need to do is provide the correct path.</P> Tue, 25 Apr 2023 14:32:58 GMT https://community.databricks.com/t5/data-engineering/spark-sparkcontext-setcheckpointdir-external-azure-storage/m-p/5312#M1763 -werners- 2023-04-25T14:32:58Z