topic Re: Overwriting the existing table in Databricks; Mechanism and History? in Data Engineering

Overwriting the existing table in Databricks; Mechanism and History?

Mado — Thu, 09 Mar 2023 03:53:06 GMT

Hi,

Assume that I have a delta table stored on an Azure storage account. When new records arrive, I repeat the transformation and overwrite the existing table.

    (DF.write
 
         .format("delta")
 
         .mode("overwrite")
 
         .option("path", save_path)
 
         .save()

I have 2 questions in this regard:

1. What is the mechanism of overwriting?

Does it truncate the table and insert new records?

2. If any overwriting operation fails, how can I know that?

Assume that dataset is large and overwriting cannot be identified by looking at the table records.

Is there any log or history that shows whether the latest overwrite was successful?

Re: Overwriting the existing table in Databricks; Mechanism and History?

-werners- — Thu, 09 Mar 2023 11:57:29 GMT

the overwrite will add new files, keep the old ones and in a log keeps track of what is current data and what is old data.

If the overwrite fails, you will get an error message in the spark program, and the data to be overwritten will still be the current state.

Re: Overwriting the existing table in Databricks; Mechanism and History?

Mado — Thu, 09 Mar 2023 20:45:43 GMT

Thanks.

Is it explained in the documentation? Could you share with me if there are any?