Where are default temporary checkpoint locations created for streaming queries with display command?

shavya — Wed, 19 Mar 2025 05:20:13 GMT

Hello!

I created a streaming query using Auto Loader to read data from S3 and used display command to see if the query was working. Initially, cloudFiles.includeExistingFiles was set to True, but since we have data in Glacier that needs to be retrieved before it can be read, the command failed. We did not provide a custom checkpoint location with the display command and spark.sql.streaming.checkpointLocation was set to None. Below is the code snippet -

source_s3_path = spark.conf.get("pipeline_dlt.source_s3_path") checkpoint_location = spark.conf.get("pipeline_dlt.schema_checkpoint_location") reader_options = { 'cloudFiles.format': 'parquet', 'cloudFiles.backfillInterval': '1 day', 'cloudFiles.schemaLocation': schema_checkpoint_location, 'mergeSchema': True, 'maxFilesPerTrigger': 1, } df = spark.readStream.format( 'cloudFiles' ).options( **reader_options ).load( source_s3_path ) display(df)

We then set cloudFiles.includeExistingFiles to False and re-ran the query. Below is the updated code -

source_s3_path = spark.conf.get("pipeline_dlt.source_s3_path") checkpoint_location = spark.conf.get("pipeline_dlt.schema_checkpoint_location") reader_options = { 'cloudFiles.format': 'parquet', 'cloudFiles.includeExistingFiles': False, 'cloudFiles.backfillInterval': '1 day', 'cloudFiles.schemaLocation': schema_checkpoint_location, 'mergeSchema': True, 'maxFilesPerTrigger': 1, } df = spark.readStream.format( 'cloudFiles' ).options( **reader_options ).load( source_s3_path ) display(df)

Even then, the command failed. It keeps looking for older files that cannot be downloaded and fails. So we surmised that the new code snippet is picking up the temporary checkpoint that Databricks creates by default.

So my question is, where does Databricks create the temporary checkpoint location by default, when none is provided? I would like to find this location and clean it up so I can run my code.

Thanks!

Re: Where are default temporary checkpoint locations created for streaming queries with display comm

Saritha_S — Wed, 11 Jun 2025 17:26:26 GMT

Hi @shavya

Good day!!

When you do not specify a checkpointLocation in a streaming query in Databricks.

It uses a temporary system directory such as:

dbfs:/local_disk0/tmp/temporary-<random_uuid>

To remove the temporary checkpoint, please set the below configuration to true.

spark.sql.streaming.forceDeleteTempCheckpointLocation true

When set to true

Spark automatically deletes the temporary checkpoint directory after the streaming query is stopped or completed.
This is useful to avoid cluttering /tmp or the Spark local directories with leftover checkpoint files.

Reference doc: https://spark.apache.org/docs/latest/configuration.html#:~:text=spark.sql.streaming.forceDeleteTempCheckpointLocation

Kindly let me know if you have any questions on this.

topic Where are default temporary checkpoint locations created for streaming queries with display command? in Data Engineering

Where are default temporary checkpoint locations created for streaming queries with display command?

Re: Where are default temporary checkpoint locations created for streaming queries with display comm