topic Re: Getting a key mismatch error with Delta Live Tables. in Data Engineering

Getting a key mismatch error with Delta Live Tables.

logan0015 — Tue, 20 Sep 2022 21:30:51 GMT

I am attempting to create a streaming delta live table. The main issue I am experiencing is the error below.

com.databricks.sql.cloudfiles.errors.CloudFilesIllegalStateException: Found mismatched event: key

I have an aws appflow that is creating a folder in s3 that is filled with parquet files. When I ingest this folder the first time everything works perfectly. However the next time the flow runs and creates a folder (whose name is unique each run) I receive the above error with it stating that the folders prefix doesn't match. Has anyone else had this issue?

Re: Getting a key mismatch error with Delta Live Tables.

logan0015 — Fri, 23 Sep 2022 07:26:20 GMT

@Debayan Mukherjee Hello! I have gone through the documentation and used it when I set up the dlt. To be more specific when I run an initial load everything works just fine. I run my appflow which creates a folder in s3 and the path would be something like s3://mybucket/ingestion/load1

Then when I run the appflow again to get the latest updates it will create a new folder in the same path. For example: s3://mybucket/ingestion/load2 that is what the error would say

com.databricks.sql.cloudfiles.errors.CloudFilesIllegalStateException: Found mismatched event: key s3://mybucket/ingestion/load2 does not have the prefix load1

Re: Getting a key mismatch error with Delta Live Tables.

Anonymous — Sun, 02 Oct 2022 11:43:31 GMT

Hi @Logan Nicol

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.

We'd love to hear from you.

Thanks!

Re: Getting a key mismatch error with Delta Live Tables.

logan0015 — Mon, 03 Oct 2022 14:01:11 GMT

Hello, @Vidula Khanna. No my issue has not been resolved. I was only pointed to the documentation I had already been using.

Re: Getting a key mismatch error with Delta Live Tables.

logan0015 — Wed, 26 Oct 2022 14:31:52 GMT

If anyone comes back to this. I ended up finding the solution on my own. DLT makes it so if you are streaming files from a location then the folder cannot change. You must drop your files into the same folder. Otherwise it complains about the name of the folder not being what it expects.

Re: Getting a key mismatch error with Delta Live Tables.

VijaC_97468 — Tue, 02 May 2023 12:06:09 GMT

Hi, I am also facing the same issue, but I found nothing on the documentation to fix it.

Re: Getting a key mismatch error with Delta Live Tables.

Debayan — Fri, 23 Sep 2022 07:09:00 GMT

Hi, Could you please go through this auto loader cloud files options:

https://docs.databricks.com/ingestion/auto-loader/options.html#auto-loader-options .

This error is not so common , we have to look into it.