cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

auser85
by New Contributor III
  • 649 Views
  • 1 replies
  • 0 kudos

With AWS/Azure Autoscaling, how do we fine tune spark jobs?

With the recommended autoscaling, e.g, https://docs.databricks.com/clusters/cluster-config-best-practices.html, setting; is it possible to dynamically set a fine tuned spark job, given that the number of executors could be changing at any time?

  • 649 Views
  • 1 replies
  • 0 kudos
Latest Reply
Aviral-Bhardwaj
Esteemed Contributor III
  • 0 kudos

@Andrew Fogarty​ I would suggest you instead of dynamic add that thing in the spark cluster itself by that you can save cost

  • 0 kudos
Labels