Thankyou @VZLA for the detailed email.
I tried to run using the configuration you suggested for single node but got the same error. I tried with different configurations but all giving the same error. Even a simple configuration like this is also failing with same error.

clusterConfig={
    "new_cluster": {
        "spark_version": "15.4.x-scala2.12",
        "effective_spark_version": "15.4.x-cpu-ml-scala2.12",
        "node_type_id": "Standard_D14_v2",
        "azure_attributes": {
            "availability": "ON_DEMAND_AZURE"
        },
        "autoscale": {
            "min_workers": 5,
            "max_workers": 15
        }
    }
}

 Thanks