topic Re: Is it possible to use both `Dynamic partition overwrites` and `overwriteSchema` options when writing a DataFrame to a Delta table?" in Machine Learning

Is it possible to use both `Dynamic partition overwrites` and `overwriteSchema` options when writing a DataFrame to a Delta table?"

Thanapat_S — Thu, 08 Jun 2023 04:12:06 GMT

In my ETL case, I want to be able to adjust the table schema as needed, meaning the number of columns may increase or decrease depending on the ETL script. Additionally, I would like to use dynamic partition overwrite to avoid potential errors when using the replacewhere option.

But based on the information provided in the document "Selectively overwrite data with Delta Lake | Databricks on AWS" it seems that this functionality is not yet supported. Is there a solution for this?

I appreciate your support. 🙏

For your information:

Databricks Runtime: 11.3 LTS (includes Apache Spark 3.3.0, Scala 2.12)

Re: Is it possible to use both `Dynamic partition overwrites` and `overwriteSchema` options when writing a DataFrame to a Delta table?"

-werners- — Thu, 08 Jun 2023 08:37:09 GMT

No and I don't see how. With Dynamic Partition Overwrite, existing logical partitions for which the write does not contain data remain unchanged.

This assumes an identical schema for all partitions, which is not guaranteed with overwriteSchema.

Re: Is it possible to use both `Dynamic partition overwrites` and `overwriteSchema` options when writing a DataFrame to a Delta table?"

Vartika — Fri, 09 Jun 2023 10:26:22 GMT

Hi @Thanapat Sontayasara,

Does @Werner Stinckens's response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly? If not, would you be happy to give us more information?

Thanks!