- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
‎01-23-2022 06:51 AM
We are running multiple Databricks job via ADF. I was wondering which option out of the below is a cheaper route for databricks notebook processing from ADF. When I create a ADF linked service, which should I use to lower my cost.
- New Job Cluster option
- Existing instance pool.
Note: I am assuming interactive cluster is out of option as this is definitely more expensive.
Thanks,
Matt
- Labels:
-
Spot Instance
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
‎01-24-2022 07:48 AM
Hi, @Matt M​ - Thank you for bringing this new question to us, and thank you for your patience while we wait for other members to respond. 🙂
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
‎01-25-2022 06:19 AM
the instance pool will be cheaper if you use spot instances. But only if you size your instance pool correctly. (number of workers and scale down time)
AFAIK you cannot use spot instances for job clusters in ADF
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
‎01-25-2022 08:05 AM
I agree with your input. I would also like to add that running Prod jobs on spot instances opens us to Azure VM eviction, so tread with caution on that front.
