brickster_2018
Databricks Employee
Databricks Employee

This is working as expected. Autoscaling is not available for spark-submit jobs

Run the job as jar job instead of spark-submit

View solution in original post