cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Pricing Spot Instance vs New Job Cluster

MattM
New Contributor III

We are running multiple Databricks job via ADF. I was wondering which option out of the below is a cheaper route for databricks notebook processing from ADF. When I create a ADF linked service, which should I use to lower my cost.

  1. New Job Cluster option
  2. Existing instance pool.

Note: I am assuming interactive cluster is out of option as this is definitely more expensive.

Thanks,

Matt

1 ACCEPTED SOLUTION

Accepted Solutions

-werners-
Esteemed Contributor III

the instance pool will be cheaper if you use spot instances. But only if you size your instance pool correctly. (number of workers and scale down time)

AFAIK you cannot use spot instances for job clusters in ADF

View solution in original post

3 REPLIES 3

Anonymous
Not applicable

Hi, @Matt M​ - Thank you for bringing this new question to us, and thank you for your patience while we wait for other members to respond. 🙂

-werners-
Esteemed Contributor III

the instance pool will be cheaper if you use spot instances. But only if you size your instance pool correctly. (number of workers and scale down time)

AFAIK you cannot use spot instances for job clusters in ADF

MattM
New Contributor III

I agree with your input. I would also like to add that running Prod jobs on spot instances opens us to Azure VM eviction, so tread with caution on that front.

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.