https://community.databricks.com/t5/data-engineering/with-aws-azure-autoscaling-how-do-we-fine-tune-spark-jobs/m-p/15954#M10204 <P>@Andrew Fogarty​&nbsp;</P><P></P><P>I would suggest you instead of dynamic add that thing in the spark cluster itself by that you can save cost</P> Tue, 20 Dec 2022 14:04:42 GMT Aviral-Bhardwaj 2022-12-20T14:04:42Z https://community.databricks.com/t5/data-engineering/with-aws-azure-autoscaling-how-do-we-fine-tune-spark-jobs/m-p/15953#M10203 <P>With the recommended autoscaling, e.g, <A href="https://docs.databricks.com/clusters/cluster-config-best-practices.html" alt="https://docs.databricks.com/clusters/cluster-config-best-practices.html" target="_blank">https://docs.databricks.com/clusters/cluster-config-best-practices.html</A>, setting; is it possible to dynamically set a fine tuned spark job, given that the number of executors could be changing at any time?</P> Mon, 19 Dec 2022 14:38:01 GMT https://community.databricks.com/t5/data-engineering/with-aws-azure-autoscaling-how-do-we-fine-tune-spark-jobs/m-p/15953#M10203 auser85 2022-12-19T14:38:01Z https://community.databricks.com/t5/data-engineering/with-aws-azure-autoscaling-how-do-we-fine-tune-spark-jobs/m-p/15954#M10204 <P>@Andrew Fogarty​&nbsp;</P><P></P><P>I would suggest you instead of dynamic add that thing in the spark cluster itself by that you can save cost</P> Tue, 20 Dec 2022 14:04:42 GMT https://community.databricks.com/t5/data-engineering/with-aws-azure-autoscaling-how-do-we-fine-tune-spark-jobs/m-p/15954#M10204 Aviral-Bhardwaj 2022-12-20T14:04:42Z