Pricing Spot Instance vs New Job Cluster

[MattM]

We are running multiple Databricks job via ADF. I was wondering which option out of the below is a cheaper route for databricks notebook processing from ADF. When I create a ADF linked service, which should I use to lower my cost.

1. New Job Cluster option
2. Existing instance pool.

Note: I am assuming interactive cluster is out of option as this is definitely more expensive.

Thanks,

Matt

[Anonymous]

Hi, @Matt M​ - Thank you for bringing this new question to us, and thank you for your patience while we wait for other members to respond. 🙂

[-werners- · Accepted solution]

the instance pool will be cheaper if you use spot instances. But only if you size your instance pool correctly. (number of workers and scale down time)

AFAIK you cannot use spot instances for job clusters in ADF

[MattM]

I agree with your input. I would also like to add that running Prod jobs on spot instances opens us to Azure VM eviction, so tread with caution on that front.