Hi,
Assume that I have a delta table stored on an Azure storage account. When new records arrive, I repeat the transformation and overwrite the existing table.
(DF.write
.format("delta")
.mode("overwrite")
.option("path", save_path)
.save()
I have 2 questions in this regard:
1. What is the mechanism of overwriting?
Does it truncate the table and insert new records?
2. If any overwriting operation fails, how can I know that?
Assume that dataset is large and overwriting cannot be identified by looking at the table records.
Is there any log or history that shows whether the latest overwrite was successful?