topic DLT Serverless costs in Data Engineering

DLT Serverless costs

mbaas — Wed, 31 Jul 2024 06:33:08 GMT

I recently started checking out the serverless delta live tables. In my understanding serverless continuous jobs (with autoloader) would only do something when new files arrive. However for serverless 4 pipelines running continuously, I spend in two days almost 2000 euros. Am I doing something wrong? If not, what is the benefit of using serverless continuously against a cheap single node cluster?

Re: DLT Serverless costs

NandiniN — Thu, 01 Aug 2024 00:50:52 GMT

Hi @mbaas ,

That does not sound right. So, were you able to compare the jobs, stages and spot the difference. Are there more tasks added, or with compute perspective you do not find a difference at all but only the cost?

Also, it may be required to look at the pipelines closely to understand the reason for increased cost. Is there already an investigation support ticket open for this?

Thanks!