https://community.databricks.com/t5/data-engineering/cost-comparison-of-dlt-to-custom-pipeline/m-p/75074#M34856 <P>DLT is expensive in my opinion. I tried to run a simple notebook that just reads a parquet file into a dataframe and write it out to a cloud storage and I got an error that i hit my CPU instance limit for my azure subscription. I just gave up after that. But this was a year ago, don't know and not interested in the update on dlt</P> Thu, 20 Jun 2024 01:54:02 GMT Zume 2024-06-20T01:54:02Z https://community.databricks.com/t5/data-engineering/cost-comparison-of-dlt-to-custom-pipeline/m-p/75051#M34854 <P>Hello, our company currently has a number of custom pipelines written in python for ETL, and I want to do an evaluation of DLT to see if that will make things more efficient.&nbsp; A problem is that there is a restriction on using DLT "because it is too expensive".&nbsp; I have not been able to find information comparing the cost of DLT to a traditional pipeline, and because it is restricted, I cannot run an actual comparison.&nbsp;&nbsp;</P><P>Has anyone done this, or is anyone aware of cost comparisons I can use to either justify trying it, or to support the claim of it being comparatively expensive?</P> Wed, 19 Jun 2024 19:51:24 GMT https://community.databricks.com/t5/data-engineering/cost-comparison-of-dlt-to-custom-pipeline/m-p/75051#M34854 DavidS1 2024-06-19T19:51:24Z https://community.databricks.com/t5/data-engineering/cost-comparison-of-dlt-to-custom-pipeline/m-p/75074#M34856 <P>DLT is expensive in my opinion. I tried to run a simple notebook that just reads a parquet file into a dataframe and write it out to a cloud storage and I got an error that i hit my CPU instance limit for my azure subscription. I just gave up after that. But this was a year ago, don't know and not interested in the update on dlt</P> Thu, 20 Jun 2024 01:54:02 GMT https://community.databricks.com/t5/data-engineering/cost-comparison-of-dlt-to-custom-pipeline/m-p/75074#M34856 Zume 2024-06-20T01:54:02Z