topic Re: Autoloader cleansource option does not take any effect in Data Engineering

Autoloader cleansource option does not take any effect

janm2 — Tue, 01 Jul 2025 12:27:11 GMT

Hello everyone,

I was very keen to try out the Autoloader's new cleanSource option so we can clean up our landing folder easily.

However I found out it does not have any effect whatsoever. As I cannot create a support case I am creating this post.

A simple streaming job such as

df = ( spark .readStream .format("cloudFiles") .option("cloudFiles.format", "json") .option("cloudFiles.schemaLocation", "s3://my_bucket/data_schema") .option("cloudFiles.schemaEvolutionMode", "addNewColumns") .option("cloudFiles.inferColumnTypes", "true") .option("cloudFiles.cleanSource", "move") .option("cloudFiles.cleanSource.retentionDuration", "2 minutes") .option("cloudFiles.cleanSource.moveDestination", "s3://my_bucket/data_moved") .load("s3://my_bucket/data")) ) ( df .writeStream .format("delta") .option("checkpointLocation", "s3://my_bucket/data_checkpoint") .option("mergeSchema", "true") .queryName("abcd") .outputMode("append") .trigger(processingTime="10 seconds") .table("dev.bronze.tmp_cloud_files_testing") )

ingests the data put into s3://my_bucket/data without issues, however the data is never moved as specified. Even when waiting for hours. I've tested few files as well as thousands of files.

I've tested s3 locations, dbfs locations as well as local locations. Autoloader has permissions to write to the locations as I could easily stream into parquet files.

What I find especially suspicious is that

SELECT * FROM cloud_files_state("s3://my_bucket/data_checkpoint")

shows there is no archive_mode set for the data.

Setting cloudFiles.cleanSource to DELETE does also not do anything.

All tested on clusters running DBR 16.4 or DBR 17.

Do we have to turn this feature on somewhere or did I implement something incorrectly? I really do not know what to try next.

Thank you
Jan

Re: Autoloader cleansource option does not take any effect

szymon_dybczak — Tue, 01 Jul 2025 14:19:31 GMT

Hi @janm2 ,

Could you try to replace your code with uppercase "MOVE" (or "DELETE" depending on which you want to use)?
I know that it sounds silly, but I've encounter several times cases where case-sensitive was shooting me in face 🙂

And since you wrote that there is no archive_mode set for the data, that makes me wonder if this is due to this reason 😄

.option("cloudfiles.cleanSource", "MOVE")

Re: Autoloader cleansource option does not take any effect

janm2 — Tue, 01 Jul 2025 15:37:30 GMT

Hello, thank you for your reply.

I have tried this before and unfortunately this was not it.

Re: Autoloader cleansource option does not take any effect

Pat — Tue, 01 Jul 2025 20:54:21 GMT

Maybe a bug, here you can find same issue:

https://community.databricks.com/t5/data-engineering/autoloader-move-file-to-archive-immediately-after-processing/td-p/120692

Re: Autoloader cleansource option does not take any effect

SanthoshU — Tue, 28 Oct 2025 22:43:55 GMT

Any Solution found??

Re: Autoloader cleansource option does not take any effect

SanthoshU — Tue, 28 Oct 2025 22:44:44 GMT

Any Solution ?

Re: Autoloader cleansource option does not take any effect

awhorton — Wed, 25 Feb 2026 16:02:02 GMT

I had the same issue, which was caused by colons in the filenames. It quietly failed in the app, but log4j contained warnings like this:

26/02/20 07:11:07 WARN CleanSourceFileMover: [queryId = f0e53] Unexpected exception when cleaning: /Volumes/prod/datalake/raw/source/entity/json_files/2026-02-19_07:12:03.924349.json
java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: 2026-02-19_07:12:03.924349.json

Changing the : to - (or some other character) resolved the problem.