delete and append in delta path

[KKo]

I am deleting data from curated path based on date column and appending staged data on it on each run, using below script. My fear is, just after the delete operation, if any network issue appeared and the job stopped before it appended the staged data on curated_path. How does the delta lake (The ACID property) handles this situation, does this roll back to the previous state since it did not append the staged data?

OR how to do delete and append properly in delta lake to avoide any data loss?

Thanks in advance!!

df_curated = spark.read.format('delta').load(curated_path)

df_curated.createOrReplaceTempView("curated_view")

sqlString = "DELETE FROM curated_view WHERE Date >= "+"'{}'" .format(daysback_date)

spark.sql(sqlString)

df_staged.write.partitionBy("Year").format('delta').mode("append").save(curated_path)

[Hubert-Dudek]

Yes, Delta is ACID, so it should work ok. As an alternative, you can always consider MERGE (as I bet that you want to replace Dates that are coming with updates)

https://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/delta-merge-into

[Aviral-Bhardwaj]

thanks man