Databricks cluster pools with init scripts

radix — Sat, 05 Oct 2024 19:59:54 GMT

Ability to submit a single job with cluster pools and init scripts

for the following payload:

{ "run_name": "A multitask job run", "timeout_seconds": 86400, "tasks": [ { "task_key": "task_1", "depends_on": [], "notebook_task": { "notebook_path": "/Workspace/Users/johndoe/task_1", "source": "WORKSPACE" }, "new_cluster": { "spark_version": "15.3.x-scala2.12", "instance_pool_id": "0926-080838-lute60-pool-91skna4w", "driver_instance_pool_id": "0926-080838-lute60-pool-91skna4w", "num_workers": 1, "init_scripts": [ { "s3": { "destination": "s3://bucket_name/init_scripts/install_utils.sh" } } ] } } ] }

this endpoint

/api/2.1/jobs/runs/submit

runs the job without passing the init scripts.

while if I send the same payload to

/api/2.1/jobs/create

it creates a job that uses both cluster pools and the init scripts.

I'm using the airflow operators such as DatabricksSubmitRunOperator (or DatabricksNotebookOperator)
which both invoke the submit endpoint, so If I want to use cluster pools the init scripts suddenly don't apply

Please let me know why is this behavior happening, is it on purpose? a known limitation?

thank you

Re: Databricks cluster pools with init scripts

Walter_C — Mon, 04 Nov 2024 21:55:23 GMT

Are you still facing issues with the job run submit API endpoint?

topic Re: Databricks cluster pools with init scripts in Data Engineering

Databricks cluster pools with init scripts

Re: Databricks cluster pools with init scripts