Saritha_S
Databricks Employee
Databricks Employee

‎06-11-2025 10:26 AM

Hi @shavya 

Good day!!

When you do not specify a checkpointLocation in a streaming query in Databricks.

It uses a temporary system directory such as:

 

 
dbfs:/local_disk0/tmp/temporary-<random_uuid>
 
To remove the temporary checkpoint, please set the below configuration to true. 
 
spark.sql.streaming.forceDeleteTempCheckpointLocation true
 
When set to true

  • Spark automatically deletes the temporary checkpoint directory after the streaming query is stopped or completed.

  • This is useful to avoid cluttering /tmp or the Spark local directories with leftover checkpoint files.

Reference doc: https://spark.apache.org/docs/latest/configuration.html#:~:text=spark.sql.streaming.forceDeleteTempC...
 
Kindly let me know if you have any questions on this. 
0 Kudos