delete and append in delta path

KKo — Sat, 07 Jan 2023 04:43:58 GMT

I am deleting data from curated path based on date column and appending staged data on it on each run, using below script. My fear is, just after the delete operation, if any network issue appeared and the job stopped before it appended the staged data on curated_path. How does the delta lake (The ACID property) handles this situation, does this roll back to the previous state since it did not append the staged data?

OR how to do delete and append properly in delta lake to avoide any data loss?

Thanks in advance!!

df_curated = spark.read.format('delta').load(curated_path)

df_curated.createOrReplaceTempView("curated_view")

sqlString = "DELETE FROM curated_view WHERE Date >= "+"'{}'" .format(daysback_date)

spark.sql(sqlString)

df_staged.write.partitionBy("Year").format('delta').mode("append").save(curated_path)

Re: delete and append in delta path

Hubert-Dudek — Sat, 07 Jan 2023 12:33:11 GMT

Yes, Delta is ACID, so it should work ok. As an alternative, you can always consider MERGE (as I bet that you want to replace Dates that are coming with updates)

https://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/delta-merge-into

Re: delete and append in delta path

Aviral-Bhardwaj — Sat, 07 Jan 2023 16:01:02 GMT

thanks man

topic Re: delete and append in delta path in Data Engineering

delete and append in delta path

Re: delete and append in delta path

Re: delete and append in delta path