topic Re: In Unity Catalog repartition method issue in Data Engineering

In Unity Catalog repartition method issue

Shiva3 — Tue, 29 Oct 2024 11:44:09 GMT

We are in the process of upgrading our notebooks to Unity Catalog. Previously, I was able to write data to an external Delta table using df.repartition(8).write. Save('path'), which correctly created multiple files. However, during the upgrade, in testing phase , this approach no longer produces the expected output.

I attempted to disable auto-compaction with spark.conf.set("spark.databricks.delta.autoCompact.enabled", "false"), but the operation still results in only one Parquet file being created in S3, rather than the intended 8. I need assistance to resolve this issue with partitioning and file output after the Unity Catalog upgrade.
Please help.

Re: In Unity Catalog repartition method issue

agallard — Tue, 29 Oct 2024 16:46:57 GMT

Hi @Shiva3,

Maybe you can try this option in Delta Lake in Unity Catalog may have optimizedWrites enabled by default, which can reduce the number of files by automatically coalescing partitions during writes.

# Disable auto-compaction and optimized writes spark.conf.set("spark.databricks.delta.autoCompact.enabled", "false") spark.conf.set("spark.databricks.delta.optimizeWrite.enabled", "false")

Setting both configurations to false ensures that Delta Lake doesn’t automatically combine files or reduce partitions, allowing df.repartition(8) to retain 8 distinct files, then you can change the config again.

Try and comment!

Regards

Re: In Unity Catalog repartition method issue

Shiva3 — Wed, 27 Aug 2025 03:25:38 GMT

Thank you @agallard , it worked, you are right Unity Catalog have optimizedWrites enabled by default.