topic DataBricks Job Orchestration in PROD in Data Engineering

DataBricks Job Orchestration in PROD

ANOOP_V — Tue, 08 Mar 2022 14:01:28 GMT

Can I suggest customer about databricks Job orchestration (public preview) ? Can we use this feature in Production as well?

Re: DataBricks Job Orchestration in PROD

Hubert-Dudek — Tue, 08 Mar 2022 15:15:02 GMT

If something is in public preview you can use it in production, just things in preview are not covered by SLAs etc.

Re: DataBricks Job Orchestration in PROD

ANOOP_V — Tue, 08 Mar 2022 17:12:48 GMT

Thanks to @Hubert Dudek ! Any idea when the features like reuse a cluster across tasks in a job and restart the DAG (such that it only runs the tasks again that had previously failed). are available?

Re: DataBricks Job Orchestration in PROD

Atanu — Sun, 17 Apr 2022 17:58:05 GMT

@ANOOP V , At present, we don’t have these features.

I assume we do plan to include it by Q4 FY2022.

What is next for Multitask Jobs?

After GA we will be working on some highly-requested features during the private preview:

- Job cluster reuse: make it possible to set a job cluster that can be shared across multiple tasks within the same job.

- Task run repair and retry: make it possible to manually restart a DAG execution from a failed step.