cancel
Showing results for 
Search instead for 
Did you mean: 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results for 
Search instead for 
Did you mean: 

Is it possible to use both `Dynamic partition overwrites` and `overwriteSchema` options when writing a DataFrame to a Delta table?"

Thanapat_S
Contributor

In my ETL case, I want to be able to adjust the table schema as needed, meaning the number of columns may increase or decrease depending on the ETL script. Additionally, I would like to use dynamic partition overwrite to avoid potential errors when using the replacewhere option.

But based on the information provided in the document "Selectively overwrite data with Delta Lake | Databricks on AWS" it seems that this functionality is not yet supported. imageIs there a solution for this?

I appreciate your support. 🙏

For your information:

Databricks Runtime: 11.3 LTS (includes Apache Spark 3.3.0, Scala 2.12)

1 ACCEPTED SOLUTION

Accepted Solutions

-werners-
Esteemed Contributor III

No and I don't see how. With Dynamic Partition Overwrite, existing logical partitions for which the write does not contain data remain unchanged.

This assumes an identical schema for all partitions, which is not guaranteed with overwriteSchema.

View solution in original post

2 REPLIES 2

-werners-
Esteemed Contributor III

No and I don't see how. With Dynamic Partition Overwrite, existing logical partitions for which the write does not contain data remain unchanged.

This assumes an identical schema for all partitions, which is not guaranteed with overwriteSchema.

Vartika
Moderator
Moderator

Hi @Thanapat Sontayasara​,

Does @Werner Stinckens​'s response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly? If not, would you be happy to give us more information?

Thanks! 

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group