delete and append in delta path

[KKo]

I am deleting data from curated path based on date column and appending staged data on it on each run, using below script. My fear is, just after the delete operation, if any network issue appeared and the job stopped before it appended the staged data on curated_path. How does the delta lake (The ACID property) handles this situation, does this roll back to the previous state since it did not append the staged data?

OR how to do delete and append properly in delta lake to avoide any data loss?

Thanks in advance!!

df_curated = spark.read.format('delta').load(curated_path)

df_curated.createOrReplaceTempView("curated_view")

sqlString = "DELETE FROM curated_view WHERE Date >= "+"'{}'" .format(daysback_date)

spark.sql(sqlString)

df_staged.write.partitionBy("Year").format('delta').mode("append").save(curated_path)

[Hubert-Dudek]

Yes, Delta is ACID, so it should work ok. As an alternative, you can always consider MERGE (as I bet that you want to replace Dates that are coming with updates)

https://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/delta-merge-into

[Aviral-Bhardwaj]

thanks man

[Kaniz]

Hi @Kris Koirala​ , We haven’t heard from you since the last response from @Hubert Dudek​ , and I was checking back to see if his suggestions helped you.

Or else, If you have any solution, please do share that with the community as it can be helpful to others.

Also, Please don't forget to click on the "Select As Best" button whenever the information provided helps resolve your question.