https://community.databricks.com/t5/data-engineering/optimizewrite-takes-too-long/m-p/50435#M28809 <P>Hi ,</P><P>&nbsp;</P><P>We have a spark job write data in delta table for last 90 date partition. We have enabled&nbsp;<SPAN>spark.databricks.delta.autoCompact.enabled and&nbsp;delta.autoOptimize.optimizeWrite. Job takes 50 mins to complete. In that logic takes 12 mins and optimizewrite takes 37 mins to complete. Is any way to reduce total job time as the output per partition is 64mb file</SPAN></P><P>&nbsp;</P><P><SPAN>We are using DBT 12 .&nbsp;</SPAN></P> Sat, 04 Nov 2023 11:35:05 GMT svrdragon 2023-11-04T11:35:05Z https://community.databricks.com/t5/data-engineering/optimizewrite-takes-too-long/m-p/50435#M28809 <P>Hi ,</P><P>&nbsp;</P><P>We have a spark job write data in delta table for last 90 date partition. We have enabled&nbsp;<SPAN>spark.databricks.delta.autoCompact.enabled and&nbsp;delta.autoOptimize.optimizeWrite. Job takes 50 mins to complete. In that logic takes 12 mins and optimizewrite takes 37 mins to complete. Is any way to reduce total job time as the output per partition is 64mb file</SPAN></P><P>&nbsp;</P><P><SPAN>We are using DBT 12 .&nbsp;</SPAN></P> Sat, 04 Nov 2023 11:35:05 GMT https://community.databricks.com/t5/data-engineering/optimizewrite-takes-too-long/m-p/50435#M28809 svrdragon 2023-11-04T11:35:05Z