by
Kash
• Contributor III
- 1241 Views
- 4 replies
- 0 kudos
Hi there,I need some help creating a new cluster policy that utilizes a single spot-instnace server to complete a job. I want to set this up as a job-compute to reduce costs and also utilize 1 spot instance.The jobs I need to ETL are very short and c...
- 1241 Views
- 4 replies
- 0 kudos
Latest Reply
Hi @Avkash Kana​,Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.
3 More Replies
- 3041 Views
- 6 replies
- 3 kudos
In my pipeline I'm using Azure Data Factory to trigger Databricks notebooks as a linked serviceI want to use spot instances for my job clusters Is there a way to achieve this?I didn't find a way to do this in the GUI.Thanks for your help!Marco
- 3041 Views
- 6 replies
- 3 kudos
Latest Reply
Hi @Werner Stinckens​ ,Just a quick follow up question.Does it make sense to you that you can select the following options in Azure Data Factory?To my understanding, "cluster version", "Python Version" and the "Worker options" are defined when I crea...
5 More Replies
- 3092 Views
- 3 replies
- 4 kudos
When I run a job enabling using spot instances , I would like to know how many number of workers are using spot and how many number of workers are using on demand instances for a given job run In order to identify the spot instances we got for any...
- 3092 Views
- 3 replies
- 4 kudos
Latest Reply
You can. do it on Azure Portal using the Virtual Machines list:You will filter by either JobId tag or RunName tag (job name) and group by azure spot eviction policy or azure spot eviction type, the vm's under Stop/Deallocate and Capacity (using the 2...
2 More Replies
- 1040 Views
- 2 replies
- 2 kudos
We are having difficulties running our jobs with spot instances that get re-claimed by AWS during shuffles. Do we have any documentation / best-practices around this? We went through this article but is there anything else to keep in mind?
- 1040 Views
- 2 replies
- 2 kudos
Latest Reply
Due to the recent changes in AWS spot market place , legacy techniques like higher spot bid price (>100%) are ineffective to retain the acquired spot node and the instances can be lost in 2 minutes notice causing workloads to fail.To mitigate this, w...
1 More Replies
- 1524 Views
- 1 replies
- 0 kudos
Does the query have to be re-run from the start, or can it continue? Trying to evaluate what risk there is by using spot instances for production jobs
- 1524 Views
- 1 replies
- 0 kudos
Latest Reply
If a spot instance is reclaimed in the middle of a job, then spark will treat it as a lost worker. The spark engine will automatically retry the tasks from the lost worker on other available workers. So the query does not have to start over if indivi...
