cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Cost comparison of DLT to custom pipeline

DavidS1
New Contributor

Hello, our company currently has a number of custom pipelines written in python for ETL, and I want to do an evaluation of DLT to see if that will make things more efficient.  A problem is that there is a restriction on using DLT "because it is too expensive".  I have not been able to find information comparing the cost of DLT to a traditional pipeline, and because it is restricted, I cannot run an actual comparison.  

Has anyone done this, or is anyone aware of cost comparisons I can use to either justify trying it, or to support the claim of it being comparatively expensive?

1 REPLY 1

Zume
New Contributor II

DLT is expensive in my opinion. I tried to run a simple notebook that just reads a parquet file into a dataframe and write it out to a cloud storage and I got an error that i hit my CPU instance limit for my azure subscription. I just gave up after that. But this was a year ago, don't know and not interested in the update on dlt

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!