cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Cost comparison of DLT to custom pipeline

DavidS1
New Contributor

Hello, our company currently has a number of custom pipelines written in python for ETL, and I want to do an evaluation of DLT to see if that will make things more efficient.  A problem is that there is a restriction on using DLT "because it is too expensive".  I have not been able to find information comparing the cost of DLT to a traditional pipeline, and because it is restricted, I cannot run an actual comparison.  

Has anyone done this, or is anyone aware of cost comparisons I can use to either justify trying it, or to support the claim of it being comparatively expensive?

1 REPLY 1

Zume
New Contributor II

DLT is expensive in my opinion. I tried to run a simple notebook that just reads a parquet file into a dataframe and write it out to a cloud storage and I got an error that i hit my CPU instance limit for my azure subscription. I just gave up after that. But this was a year ago, don't know and not interested in the update on dlt

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group