topic Re: Changing default Delta behavior in DBR 8.x for writes in Data Engineering

Changing default Delta behavior in DBR 8.x for writes

Anonymous — Mon, 21 Jun 2021 21:46:41 GMT

Is there anyway to add a Spark Config that reverts the default behavior when doing tables writes from Delta to Parquet in DBR 8.0+? I know you can simply specify .format("parquet") but that could involve a decent amount of code change for some clients.

https://docs.databricks.com/release-notes/runtime/8.0-migration.html#{datasetdataframe}{writewritetowritestream}-apis

Re: Changing default Delta behavior in DBR 8.x for writes

Ryan_Chynoweth — Tue, 22 Jun 2021 22:17:48 GMT

Please check out our migration guide on this topic.

Option one would be your best bet. You can set "spark.sql.sources.default" to parquet and "spark.sql.legacy.createHiveTableByDefault" to true on your cluster.

Option two would be to change your code.

Re: Changing default Delta behavior in DBR 8.x for writes

Anonymous — Tue, 22 Jun 2021 22:26:30 GMT

Thanks @Ryan Chynoweth !