topic Re: error after updating delta table com.databricks.sql.transaction.tahoe.DeltaUnsupportedOperationException: Detected a data update in Data Engineering

error after updating delta table com.databricks.sql.transaction.tahoe.DeltaUnsupportedOperationException: Detected a data update

sanjay — Thu, 30 Mar 2023 12:02:21 GMT

Hi,

I have pipeline running. I have updated one file in delta table which is already processed. Now I am getting error

com.databricks.sql.transaction.tahoe.DeltaUnsupportedOperationException: Detected a data update. This is currently not supported. If you'd like to ignore updates, set the option 'ignoreChanges' to 'true'.

I have set ignoreChanges to true but still getting same error.

spark.readStream.format("delta")

.option("ignoreChanges", "true")

.load("/tmp/delta/user_events")

Regards,

Sanjay

Re: error after updating delta table com.databricks.sql.transaction.tahoe.DeltaUnsupportedOperationException: Detected a data update

karthik_p — Thu, 30 Mar 2023 12:16:03 GMT

@Sanjay Jain which run time version you are using, above should work if you have updated required field and that should be consumed by downstream consumers, please check below article

https://docs.databricks.com/structured-streaming/delta-lake.html#ignore-updates-and-deletes

Re: error after updating delta table com.databricks.sql.transaction.tahoe.DeltaUnsupportedOperationException: Detected a data update

sanjay — Thu, 30 Mar 2023 12:20:06 GMT

I am using 11.3 LTS. I have updated one field and this data is consumed by downstream consumer. But challenge is, I am getting other unchanged files as well to consumer along with updated file.

Re: error after updating delta table com.databricks.sql.transaction.tahoe.DeltaUnsupportedOperationException: Detected a data update

Sandeep — Thu, 30 Mar 2023 13:50:03 GMT

@Sanjay Jain how did you update the file? Can you elaborate on the steps, please?

Re: error after updating delta table com.databricks.sql.transaction.tahoe.DeltaUnsupportedOperationException: Detected a data update

sanjay — Thu, 30 Mar 2023 13:57:36 GMT

Using merge into table a

using table b

when matched then update SET a.name= b.name

Re: error after updating delta table com.databricks.sql.transaction.tahoe.DeltaUnsupportedOperationException: Detected a data update

sanjay — Fri, 31 Mar 2023 12:33:40 GMT

I can see ignoreChanges true emits all the updates + emits unupdated files as well for same partition. As per documents, need to handle duplicates in downstream. Can you suggest how to handle duplicate files.

This is from databrick documents. (https://docs.databricks.com/structured-streaming/delta-lake.html#ignore-updates-and-deletes)

"The semantics for ignoreChanges differ greatly from skipChangeCommits. With ignoreChanges enabled, rewritten data files in the source table are re-emitted after a data changing operation such as UPDATE, MERGE INTO, DELETE (within partitions), or OVERWRITE. Unchanged rows are often emitted alongside new rows, so downstream consumers must be able to handle duplicates. Deletes are not propagated downstream. ignoreChanges subsumes ignoreDeletes."

Re: error after updating delta table com.databricks.sql.transaction.tahoe.DeltaUnsupportedOperationException: Detected a data update

Anonymous — Sat, 01 Apr 2023 02:19:40 GMT

Hi @Sanjay Jain

Hope everything is going great.

Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we can help you.

Cheers!

Re: error after updating delta table com.databricks.sql.transaction.tahoe.DeltaUnsupportedOperationException: Detected a data update

sanjay — Mon, 03 Apr 2023 07:01:22 GMT

Hi Vidula,

Sorry I am still looking for solution. Appreciate if you can provide any help.

Regards,

Sanjay

Re: error after updating delta table com.databricks.sql.transaction.tahoe.DeltaUnsupportedOperation

Sanjeev_Chauhan — Mon, 01 Jan 2024 20:10:24 GMT

Hi Sanjay,
You can try adding .option("overwriteSchema", "true")