topic Re: Issue with Autoloader cleanSource=MOVE Not Working as Expected in Data Engineering

Issue with Autoloader cleanSource=MOVE Not Working as Expected

nikhilshetty4 — Fri, 22 Aug 2025 10:01:09 GMT

Hi everyone,

I've been trying to explore on cleanSource option in Autoloader to move files from the source to an archive location after they're processed and loaded into a table. I used the following simple code to test this functionality. While the code executes without any errors, the files remain in the source location and are not moved.

source_path = "abfss://container@storage_acc.dfs.core.windows.net/source/files"

archive_path = "abfss://container@storage_acc.dfs.core.windows.net/archive/files"

schema_location = "abfss://container@storage_acc.dfs.core.windows.net/source/autoloader/schema"

checkpoint_location = "abfss://container@storage_acc.dfs.core.windows.net/source/autoloader/checkpoint"

df = (

spark.readStream

.format("cloudFiles")

.option("cloudFiles.format", "parquet")

.option("cloudFiles.schemaLocation", schema_location)

.option("cloudFiles.cleanSource", "MOVE")

.option("cloudFiles.cleanSource.moveDestination", archive_path)

.option("cloudFiles.includeExistingFiles", "true")

.load(source_path))

)

df.writeStream

.format("delta")

.option("checkpointLocation", checkpoint_location)

.outputMode("append")

.table("uc.schema.table_name")

When I run the query SELECT * FROM cloud_files_state(checkpoint_location), I notice that the archive_mode and move_location columns are NULL, even though I’ve explicitly set cleanSource to MOVE. I also tested the DELETE option with .option("cloudFiles.cleanSource.retentionDuration", "7 days"), but that didn’t work either.

I came across a similar issue reported by another user using an S3 bucket as the source: Autoloader cleansource option does not take any effect

I’ve tested this both in a notebook with a cluster running on Runtime 17.0 and using DLT with Runtime 16.4.

Could someone help me understand if I’m missing something or if there are any prerequisites or configurations needed to make this work?

Thanks,

Nikhil

Re: Issue with Autoloader cleanSource=MOVE Not Working as Expected

szymon_dybczak — Fri, 22 Aug 2025 10:21:35 GMT

Hi @nikhilshetty4 ,

I think it might be some kind of bug related to that feautre. You are another person who is saying that it doesn't work as expected

Autoloader move file to archive immediately after ... - Databricks Community - 120692

Re: Issue with Autoloader cleanSource=MOVE Not Working as Expected

Advika — Fri, 22 Aug 2025 11:19:52 GMT

Hello @nikhilshetty4!

To confirm, do the files show a non-null commit_time in cloud_files_state? They’ll only move to the archive location after this is set and the retention period has elapsed.

Re: Issue with Autoloader cleanSource=MOVE Not Working as Expected

nikhilshetty4 — Fri, 22 Aug 2025 11:34:52 GMT

Hi @Advika
Yes, the commit_time column contains valid timestamp values. However, the archive_time, archive_mode, and move_location columns are all showing null.

Re: Issue with Autoloader cleanSource=MOVE Not Working as Expected

Advika — Fri, 22 Aug 2025 12:08:08 GMT

@nikhilshetty4, If the archive columns (archive_time, archive_mode, move_location) are null, it means files haven’t been picked up by cleanSource for move/delete yet. Move/delete occurs after commit_time is set, the retention period has passed, and the stream is actively processing. If the stream is stopped, cleanup won’t occur, it resumes the next time the stream runs and processes data.

Re: Issue with Autoloader cleanSource=MOVE Not Working as Expected

nikhilshetty4 — Fri, 22 Aug 2025 12:45:29 GMT

@Advika, I’ve tried setting the retention duration to 1 or 2 minutes and kept the stream running well beyond that time. Even when new files were processed during the stream, the data still wasn’t moved to the archive location.
I've attached the screenshot of cloud_files_state output:

I did see that there is no time restriction for MOVE in autoloader documentation:

Re: Issue with Autoloader cleanSource=MOVE Not Working as Expected

Advika — Fri, 22 Aug 2025 13:05:34 GMT

Thanks for sharing the details, @nikhilshetty4.
I recommend raising a case with the Databricks Support team and including all the relevant details. This will help them investigate and resolve the issue more quickly.

Re: Issue with Autoloader cleanSource=MOVE Not Working as Expected

nikhilshetty4 — Fri, 22 Aug 2025 13:32:19 GMT

Got it, Thanks! I'll raise a case with Databricks Support team.

Re: Issue with Autoloader cleanSource=MOVE Not Working as Expected

Ozear — Fri, 26 Sep 2025 16:29:13 GMT

any update on this ?

Re: Issue with Autoloader cleanSource=MOVE Not Working as Expected

kunalmishra9 — Tue, 10 Feb 2026 20:45:58 GMT

We're having the same issue, where for a continuous-trigger on a long-running stream, it took 10 hours to "MOVE" files out of the directory for a retention period of 30s. Ideally, we'd want to move stuff out immediately, since incoming files slow down the autoloader in directory listing mode (yes, planning on moving to file notifications ASAP, but can't dedicate time to it while blocking a launch).