https://community.databricks.com/t5/data-engineering/thank-you-for-the-quot-setting-up-tables-quot-speed-up-from-dlt/m-p/101510#M40702 <P>Hi&nbsp;<a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/122949">@charl-p-botha</a>&nbsp;Thanks for your feedback !</P> <P>Thank you for sharing your experience with Delta Live Tables (DLT) performance improvements. We are pleased to hear that the upgrade to dlt:15.4.4-delta-pipelines-dlt-release-2024.44-rc1-commit-1a62345-image-ba3b9ec has significantly reduced the 'setting up tables' stage time for your 300 tables case from 27 minutes to 14 minutes.</P> <P>We appreciate your positive feedback and want to assure you that we are continuously working on further optimizations to enhance performance.&nbsp;</P> <P>Thank you for your support and for being a valued member of our community. If you have any further questions or feedback, please do not hesitate to reach out.</P> Mon, 09 Dec 2024 17:44:33 GMT VZLA 2024-12-09T17:44:33Z https://community.databricks.com/t5/data-engineering/thank-you-for-the-quot-setting-up-tables-quot-speed-up-from-dlt/m-p/98041#M39591 <P>Dear DataBricks people,</P><P>We are currently measuring DLT performance and cost on a medallion architecture with 150 to 300 tables, and we're interested in adding even more tables.</P><P>I've been doing automated incremental streaming DLT pipelines every 3 hours through the night, and was pleasantly surprised this morning to see that the "setting up tables" (aka SETTING_UP_TABLES) stage for the 300 tables case went from 27 minutes down to 14 minutes when the preview runtime upgraded from `dlt:15.4.4-delta-pipelines-dlt-release-2024.42-rc0-commit-10aaba0-image-c48da6f` to `dlt:15.4.4-delta-pipelines-dlt-release-2024.44-rc1-commit-1a62345-image-ba3b9ec`.</P><P>Thank you for this already substantial speed-up, but please do keep on improving it if at all possible!</P><P>I also hope that you have some good performance testing in your regression testing suite.</P><P>Relevant previous thread on this forum:&nbsp;<A href="https://community.databricks.com/t5/data-engineering/delta-live-tables-too-much-time-to-do-the-quot-setting-up-quot/m-p/97561#M39503" target="_blank" rel="noopener">https://community.databricks.com/t5/data-engineering/delta-live-tables-too-much-time-to-do-the-quot-setting-up-quot/m-p/97561#M39503</A></P> Thu, 07 Nov 2024 07:15:48 GMT https://community.databricks.com/t5/data-engineering/thank-you-for-the-quot-setting-up-tables-quot-speed-up-from-dlt/m-p/98041#M39591 charl-p-botha 2024-11-07T07:15:48Z https://community.databricks.com/t5/data-engineering/thank-you-for-the-quot-setting-up-tables-quot-speed-up-from-dlt/m-p/101510#M40702 <P>Hi&nbsp;<a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/122949">@charl-p-botha</a>&nbsp;Thanks for your feedback !</P> <P>Thank you for sharing your experience with Delta Live Tables (DLT) performance improvements. We are pleased to hear that the upgrade to dlt:15.4.4-delta-pipelines-dlt-release-2024.44-rc1-commit-1a62345-image-ba3b9ec has significantly reduced the 'setting up tables' stage time for your 300 tables case from 27 minutes to 14 minutes.</P> <P>We appreciate your positive feedback and want to assure you that we are continuously working on further optimizations to enhance performance.&nbsp;</P> <P>Thank you for your support and for being a valued member of our community. If you have any further questions or feedback, please do not hesitate to reach out.</P> Mon, 09 Dec 2024 17:44:33 GMT https://community.databricks.com/t5/data-engineering/thank-you-for-the-quot-setting-up-tables-quot-speed-up-from-dlt/m-p/101510#M40702 VZLA 2024-12-09T17:44:33Z