topic Re: Pricing Spot Instance vs New Job Cluster in Data Engineering

Pricing Spot Instance vs New Job Cluster

MattM — Sun, 23 Jan 2022 14:51:30 GMT

We are running multiple Databricks job via ADF. I was wondering which option out of the below is a cheaper route for databricks notebook processing from ADF. When I create a ADF linked service, which should I use to lower my cost.

New Job Cluster option
Existing instance pool.

Note: I am assuming interactive cluster is out of option as this is definitely more expensive.

Thanks,

Matt

Re: Pricing Spot Instance vs New Job Cluster

Anonymous — Mon, 24 Jan 2022 15:48:22 GMT

Hi, @Matt M - Thank you for bringing this new question to us, and thank you for your patience while we wait for other members to respond. 🙂

Re: Pricing Spot Instance vs New Job Cluster

-werners- — Tue, 25 Jan 2022 14:19:25 GMT

the instance pool will be cheaper if you use spot instances. But only if you size your instance pool correctly. (number of workers and scale down time)

AFAIK you cannot use spot instances for job clusters in ADF

Re: Pricing Spot Instance vs New Job Cluster

MattM — Tue, 25 Jan 2022 16:05:25 GMT

I agree with your input. I would also like to add that running Prod jobs on spot instances opens us to Azure VM eviction, so tread with caution on that front.