Databricks cluster pools with init scripts

[radix]

Ability to submit a single job with cluster pools and init scripts

for the following payload:

{
    "run_name": "A multitask job run",
    "timeout_seconds": 86400,
    "tasks": [
        {
            "task_key": "task_1",
            "depends_on": [],
            "notebook_task": {
                "notebook_path": "/Workspace/Users/johndoe/task_1",
                "source": "WORKSPACE"
            },
            "new_cluster": {
                "spark_version": "15.3.x-scala2.12",
                "instance_pool_id": "0926-080838-lute60-pool-91skna4w",
                "driver_instance_pool_id": "0926-080838-lute60-pool-91skna4w",
                "num_workers": 1,
                "init_scripts": [
                    {
                        "s3": {
                            "destination": "s3://bucket_name/init_scripts/install_utils.sh"
                        }
                    }
                ]
            }
        }
    ]
}

this endpoint

/api/2.1/jobs/runs/submit

 runs the job without passing the init scripts.

while if I send the same payload to 

/api/2.1/jobs/create

 it creates a job that uses both cluster pools and the init scripts.

I'm using the airflow operators such as  DatabricksSubmitRunOperator (or DatabricksNotebookOperator)
which both invoke the submit endpoint, so If I want to use cluster pools the init scripts suddenly don't apply

Please let me know why is this behavior happening, is it on purpose? a known limitation?

thank you