Masoomeh
Databricks Partner

I followed @arpit suggestion and set the cluster configuration explicitly in the JSON file and solved the issue.