cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
auser85
by • New Contributor III
  • 1036 Views
  • 1 replies
  • 0 kudos

With AWS/Azure Autoscaling, how do we fine tune spark jobs?

With the recommended autoscaling, e.g, https://docs.databricks.com/clusters/cluster-config-best-practices.html, setting; is it possible to dynamically set a fine tuned spark job, given that the number of executors could be changing at any time?

Data Engineering
Reply
  • 1036 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Aviral-Bhardwaj
Esteemed Contributor III
  • 0 kudos

@Andrew Fogarty​ I would suggest you instead of dynamic add that thing in the spark cluster itself by that you can save cost

  • 0 kudos
Labels