Hi SamAdams,

How are doing today? , Really appreciate you sharing what worked and what didn’t in your case! You're absolutely right—when switching buckets, not just folders within a bucket, that spark.databricks.cloudFiles.checkSourceChanged config becomes critical to avoid that StreamingQueryException. Also great point about the includeExistingFiles=False only applying on the very first run—it's one of those gotchas that can easily catch folks off guard when doing incremental updates or migrations. Doing a full reset and isolating the bronze layer with the pipelines.reset.allowed = false property is a smart call, especially when you want to preserve downstream processing. And yeah, pre-copying to a separate folder and moving it in when ready is a nice, clean strategy too. Thanks again for sharing—this will definitely help others running into similar challenges!

Regards,

Brahma