- 4369 Views
- 5 replies
- 1 kudos
Autoloader and "cleanSource"
Hi All, We are trying to use the Spark 3 structured streaming feature/option ".option('cleanSource','archive')" to archive processed files. This is working as expected using the standard spark implementation, however does not appear to work using aut...
- 4369 Views
- 5 replies
- 1 kudos
Latest Reply
- 1 kudos
https://docs.databricks.com/ingestion/auto-loader/options.html#common-auto-loader-optionscleanSource is not a listed option so it won't do anything.Maybe event retention is something you can use?
- 1 kudos