Where are default temporary checkpoint locations created for streaming queries with display command?

[shavya]

Hello!

I created a streaming query using Auto Loader to read data from S3 and used display command to see if the query was working. Initially, cloudFiles.includeExistingFiles was set to True, but since we have data in Glacier that needs to be retrieved before it can be read, the command failed. We did not provide a custom checkpoint location with the display command and spark.sql.streaming.checkpointLocation was set to None. Below is the code snippet -

source_s3_path = spark.conf.get("pipeline_dlt.source_s3_path")
checkpoint_location = spark.conf.get("pipeline_dlt.schema_checkpoint_location")

reader_options = {
    'cloudFiles.format': 'parquet',
    'cloudFiles.backfillInterval': '1 day',
    'cloudFiles.schemaLocation': schema_checkpoint_location,
    'mergeSchema': True,
    'maxFilesPerTrigger': 1,
}

df = spark.readStream.format(
    'cloudFiles'
).options(
    **reader_options
).load(
    source_s3_path
)

display(df)

We then set cloudFiles.includeExistingFiles to False and re-ran the query. Below is the updated code -

source_s3_path = spark.conf.get("pipeline_dlt.source_s3_path")
checkpoint_location = spark.conf.get("pipeline_dlt.schema_checkpoint_location")

reader_options = {
    'cloudFiles.format': 'parquet',
    'cloudFiles.includeExistingFiles': False,
    'cloudFiles.backfillInterval': '1 day',
    'cloudFiles.schemaLocation': schema_checkpoint_location,
    'mergeSchema': True,
    'maxFilesPerTrigger': 1,
}

df = spark.readStream.format(
    'cloudFiles'
).options(
    **reader_options
).load(
    source_s3_path
)

display(df)

Even then, the command failed. It keeps looking for older files that cannot be downloaded and fails. So we surmised that the new code snippet is picking up the temporary checkpoint that Databricks creates by default.

So my question is, where does Databricks create the temporary checkpoint location by default, when none is provided? I would like to find this location and clean it up so I can run my code.

Thanks!