cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Overwriting the existing table in Databricks; Mechanism and History?

Mado
Valued Contributor II

Hi,

Assume that I have a delta table stored on an Azure storage account. When new records arrive, I repeat the transformation and overwrite the existing table.

    (DF.write
 
         .format("delta")
 
         .mode("overwrite")
 
         .option("path", save_path)
 
         .save()

I have 2 questions in this regard:

1. What is the mechanism of overwriting?

Does it truncate the table and insert new records?

2. If any overwriting operation fails, how can I know that?

Assume that dataset is large and overwriting cannot be identified by looking at the table records.

Is there any log or history that shows whether the latest overwrite was successful?

2 REPLIES 2

-werners-
Esteemed Contributor III

the overwrite will add new files, keep the old ones and in a log keeps track of what is current data and what is old data.

If the overwrite fails, you will get an error message in the spark program, and the data to be overwritten will still be the current state.

Mado
Valued Contributor II

Thanks.

Is it explained in the documentation? Could you share with me if there are any?

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group