DataBricks Job Orchestration in PROD

[ANOOP_V]

Can I suggest customer about databricks Job orchestration (public preview) ? Can we use this feature in Production as well?

[Hubert-Dudek]

If something is in public preview you can use it in production, just things in preview are not covered by SLAs etc.

[ANOOP_V]

Thanks to @Hubert Dudek​ ! Any idea when the features like reuse a cluster across tasks in a job and restart the DAG (such that it only runs the tasks again that had previously failed). are available?

[Atanu · Accepted solution]

@ANOOP V​ , At present, we don’t have these features.

I assume we do plan to include it by Q4 FY2022.

What is next for Multitask Jobs?

After GA we will be working on some highly-requested features during the private preview:

- Job cluster reuse: make it possible to set a job cluster that can be shared across multiple tasks within the same job.

- Task run repair and retry: make it possible to manually restart a DAG execution from a failed step.