saurabh18cs
Honored Contributor III

how about mounting cloud storage?

spark.conf.set("spark.sql.streaming.checkpointLocation", "dbfs:/mnt/your-checkpoint-directory")