Getting a key mismatch error with Delta Live Tables.

[logan0015]

I am attempting to create a streaming delta live table. The main issue I am experiencing is the error below.

com.databricks.sql.cloudfiles.errors.CloudFilesIllegalStateException: Found mismatched event: key

I have an aws appflow that is creating a folder in s3 that is filled with parquet files. When I ingest this folder the first time everything works perfectly. However the next time the flow runs and creates a folder (whose name is unique each run) I receive the above error with it stating that the folders prefix doesn't match. Has anyone else had this issue?

[Debayan]

Hi, Could you please go through this auto loader cloud files options:

https://docs.databricks.com/ingestion/auto-loader/options.html#auto-loader-options .

This error is not so common , we have to look into it.

[logan0015]

@Debayan Mukherjee​ Hello! I have gone through the documentation and used it when I set up the dlt. To be more specific when I run an initial load everything works just fine. I run my appflow which creates a folder in s3 and the path would be something like s3://mybucket/ingestion/load1

Then when I run the appflow again to get the latest updates it will create a new folder in the same path. For example: s3://mybucket/ingestion/load2 that is what the error would say

com.databricks.sql.cloudfiles.errors.CloudFilesIllegalStateException: Found mismatched event: key s3://mybucket/ingestion/load2 does not have the prefix load1

[Anonymous]

Hi @Logan Nicol​ 

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. 

We'd love to hear from you.

Thanks!

[logan0015]

Hello, @Vidula Khanna​. No my issue has not been resolved. I was only pointed to the documentation I had already been using.

[logan0015 · Accepted solution]

If anyone comes back to this. I ended up finding the solution on my own. DLT makes it so if you are streaming files from a location then the folder cannot change. You must drop your files into the same folder. Otherwise it complains about the name of the folder not being what it expects.

[VijaC_97468]

Hi, I am also facing the same issue, but I found nothing on the documentation to fix it.