Use Spot Instances with Azure Data Factory Linked Service

[MarcoCaviezel]

In my pipeline I'm using Azure Data Factory to trigger Databricks notebooks as a linked service

I want to use spot instances for my job clusters

Is there a way to achieve this?

I didn't find a way to do this in the GUI.

Thanks for your help!

Marco

[Kaniz]

Hi @ MarcoCaviezel! My name is Kaniz, and I'm the technical moderator here. Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question first. Or else I will get back to you soon. Thanks.

[Anonymous]

@Marco Caviezel​ - Does the information in this thread, Spot instance in Azure Databricks, answer your question?

[-werners-]

Hi Marco,

the easiest way to do this is using a pool which you defined as 'All Spot'.

In the pool definition you can set 0 as 'min idle' to avoid nodes doing nothing (if you do not want warm instances).

I expect MS to add a 'spot' selector in ADF though, without the need to work with pools.

[MarcoCaviezel · Accepted solution]

Hi @Werner Stinckens​ ,

Thanks a lot for your helpful information!

Until now I never worked with pools but after taking a closer look at it I can profit from a lot of other advantages on top of the "All Spot" option.

Have a great day!

Cheers,

Marco

[MarcoCaviezel]

Hi @Werner Stinckens​ ,

Just a quick follow up question.

Does it make sense to you that you can select the following options in Azure Data Factory?

To my understanding, "cluster version", "Python Version" and the "Worker options" are defined when I create the cluster in Databricks.

Thanks a lot for your help!

Marco

[-werners-]

It does not make sense indeed as the pool decides this. I think this is still WIP in ADF.

I select the same the python version as the pool, and make sure the autoscaling does not exceed the max nodes.