09-20-2022 02:30 PM
I am attempting to create a streaming delta live table. The main issue I am experiencing is the error below.
com.databricks.sql.cloudfiles.errors.CloudFilesIllegalStateException: Found mismatched event: key
I have an aws appflow that is creating a folder in s3 that is filled with parquet files. When I ingest this folder the first time everything works perfectly. However the next time the flow runs and creates a folder (whose name is unique each run) I receive the above error with it stating that the folders prefix doesn't match. Has anyone else had this issue?
10-26-2022 07:31 AM
If anyone comes back to this. I ended up finding the solution on my own. DLT makes it so if you are streaming files from a location then the folder cannot change. You must drop your files into the same folder. Otherwise it complains about the name of the folder not being what it expects.
09-23-2022 12:09 AM
Hi, Could you please go through this auto loader cloud files options:
https://docs.databricks.com/ingestion/auto-loader/options.html#auto-loader-options .
This error is not so common , we have to look into it.
09-23-2022 12:26 AM
@Debayan Mukherjee Hello! I have gone through the documentation and used it when I set up the dlt. To be more specific when I run an initial load everything works just fine. I run my appflow which creates a folder in s3 and the path would be something like s3://mybucket/ingestion/load1
Then when I run the appflow again to get the latest updates it will create a new folder in the same path. For example: s3://mybucket/ingestion/load2 that is what the error would say
com.databricks.sql.cloudfiles.errors.CloudFilesIllegalStateException: Found mismatched event: key s3://mybucket/ingestion/load2 does not have the prefix load1
10-02-2022 04:43 AM
Hi @Logan Nicol
Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.
We'd love to hear from you.
Thanks!
10-03-2022 07:01 AM
Hello, @Vidula Khanna. No my issue has not been resolved. I was only pointed to the documentation I had already been using.
10-26-2022 07:31 AM
If anyone comes back to this. I ended up finding the solution on my own. DLT makes it so if you are streaming files from a location then the folder cannot change. You must drop your files into the same folder. Otherwise it complains about the name of the folder not being what it expects.
05-02-2023 05:06 AM
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group