topic Re: Use Spot Instances with Azure Data Factory Linked Service in Data Engineering

Use Spot Instances with Azure Data Factory Linked Service

MarcoCaviezel — Thu, 07 Oct 2021 09:32:49 GMT

In my pipeline I'm using Azure Data Factory to trigger Databricks notebooks as a linked service

I want to use spot instances for my job clusters

Is there a way to achieve this?

I didn't find a way to do this in the GUI.

Thanks for your help!

Marco

Anonymous — Thu, 07 Oct 2021 17:13:08 GMT

@Marco Caviezel - Does the information in this thread, Spot instance in Azure Databricks, answer your question?

-werners- — Fri, 08 Oct 2021 06:55:00 GMT

Hi Marco,

the easiest way to do this is using a pool which you defined as 'All Spot'.

In the pool definition you can set 0 as 'min idle' to avoid nodes doing nothing (if you do not want warm instances).

I expect MS to add a 'spot' selector in ADF though, without the need to work with pools.

MarcoCaviezel — Mon, 11 Oct 2021 06:22:27 GMT

Hi @Werner Stinckens ,

Thanks a lot for your helpful information!

Until now I never worked with pools but after taking a closer look at it I can profit from a lot of other advantages on top of the "All Spot" option.

Have a great day!

Cheers,

Marco

MarcoCaviezel — Mon, 11 Oct 2021 14:19:14 GMT

Hi @Werner Stinckens ,

Just a quick follow up question.

Does it make sense to you that you can select the following options in Azure Data Factory?

To my understanding, "cluster version", "Python Version" and the "Worker options" are defined when I create the cluster in Databricks.

Thanks a lot for your help!

Marco

-werners- — Mon, 11 Oct 2021 14:44:26 GMT

It does not make sense indeed as the pool decides this. I think this is still WIP in ADF.

I select the same the python version as the pool, and make sure the autoscaling does not exceed the max nodes.