cancel
Showing results for 
Search instead for 
Did you mean: 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results for 
Search instead for 
Did you mean: 

Is it possible to use both `Dynamic partition overwrites` and `overwriteSchema` options when writing a DataFrame to a Delta table?"

Thanapat_S
Contributor

In my ETL case, I want to be able to adjust the table schema as needed, meaning the number of columns may increase or decrease depending on the ETL script. Additionally, I would like to use dynamic partition overwrite to avoid potential errors when using the replacewhere option.

But based on the information provided in the document "Selectively overwrite data with Delta Lake | Databricks on AWS" it seems that this functionality is not yet supported. imageIs there a solution for this?

I appreciate your support. 🙏

For your information:

Databricks Runtime: 11.3 LTS (includes Apache Spark 3.3.0, Scala 2.12)

1 ACCEPTED SOLUTION

Accepted Solutions

-werners-
Esteemed Contributor III

No and I don't see how. With Dynamic Partition Overwrite, existing logical partitions for which the write does not contain data remain unchanged.

This assumes an identical schema for all partitions, which is not guaranteed with overwriteSchema.

View solution in original post

2 REPLIES 2

-werners-
Esteemed Contributor III

No and I don't see how. With Dynamic Partition Overwrite, existing logical partitions for which the write does not contain data remain unchanged.

This assumes an identical schema for all partitions, which is not guaranteed with overwriteSchema.

Vartika
Moderator
Moderator

Hi @Thanapat Sontayasara​,

Does @Werner Stinckens​'s response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly? If not, would you be happy to give us more information?

Thanks! 

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!