CloudFilesIllegalStateException: Found mismatched event: key old_file_path doesn't have the prefix: new_file_path

[lurban]

My team currently uses Autoloader and Delta Live Tables to process incremental data from ADLS storage. We are needing to keep the same table and history, but switch the filepath to a different location in storage. When I test a filepath change, I receive this error in DLT with OLD_FILE_PATH being the original directory and NEW_FILE_PATH being the new directory:

com.databricks.sql.cloudfiles.errors.CloudFilesIllegalStateException: Found mismatched event: key OLD_FILE_PATH doesn't have the prefix: NEW_FILE_PATH

Has anyone else come across an error like this while needing to change file paths with Autoloader/DLT? Are there recommendations for how to successfully make this switch without resetting and losing the current data?

Thanks in advance!

[DD_Sharma]

Autoloader doesn't support changing the source path for running job so if you change your source path your stream fails because the source path has changed. However, if you really want to change the path you can change it by using the new checkpoint directory. But that makes it difficult to perform targeted backfills and all.