@vivek_cloudde thanks for your question!

You can absolutely create an on-demand cluster with auto-scaling on Azure—it’s not blocked. You just need to remove any conflicting parameters so the API sees only one setting for worker configuration.

You’re hitting the error because Databricks expects exactly one of these in your cluster config: num_workers, or
autoscale, or virtual_cluster_size (rarely used). So, you can have an on-demand cluster ("availability": "ON_DEMAND_AZURE") with autoscaling by specifying:

"autoscale": {
  "min_workers": 2,
  "max_workers": 8
}

Although in this case, the error message could be confusing. Now, coming back to your JSON file, nothing in this snippet explicitly sets autoscale or min_workers field, it only has "num_workers": 2, so the error suggests something else is adding or conflicting with autoscale / virtual_cluster_size. Is this the actual JSON going to the API endpoint?

The only unusual thing I can spot on this JSON, is that you have "SingleNode" in custom_tags, but you're also specifying num_workers. I'm not very confident this would work, a SingleNode, usually doesn't have workers? or num_workers would be 0 ? But probably even setting num_workers regardless of its value the assertion would still fail to validate? Please try removing num_workers first.

Can you please try with the following, or something simpler, and start building it up while fixing the issues along the way:

clusterConfig={
    "new_cluster": {
        "spark_version": "15.4.x-scala2.12",
        "node_type_id": "Standard_D14_v2",
        "spark_conf": {
            "spark.databricks.cluster.profile": "singleNode",
            "spark.master": "local[*]",
            "spark.databricks.delta.preview.enabled": true
        },
        "custom_tags": {
            "ResourceClass": "SingleNode"
        },
        "azure_attributes": {
            "first_on_demand": 1,
            "availability": "ON_DEMAND_AZURE",
            "spot_bid_max_price": "-1"
        },
        "enable_elastic_disk": true,
        "num_workers": 0,
        "autotermination_minutes": 10
    }
}