cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

DataBricks Job Orchestration in PROD

ANOOP_V
New Contributor II

Can I suggest customer about databricks Job orchestration (public preview) ? Can we use this feature in Production as well?

1 ACCEPTED SOLUTION

Accepted Solutions

Atanu
Databricks Employee
Databricks Employee

@ANOOP V​ , At present, we don’t have these features.

I assume we do plan to include it by Q4 FY2022.

What is next for Multitask Jobs?

After GA we will be working on some highly-requested features during the private preview:

- Job cluster reuse: make it possible to set a job cluster that can be shared across multiple tasks within the same job.

- Task run repair and retry: make it possible to manually restart a DAG execution from a failed step.

View solution in original post

3 REPLIES 3

Hubert-Dudek
Esteemed Contributor III

If something is in public preview you can use it in production, just things in preview are not covered by SLAs etc.

ANOOP_V
New Contributor II

Thanks to @Hubert Dudek​ ! Any idea when the features like reuse a cluster across tasks in a job and restart the DAG (such that it only runs the tasks again that had previously failed). are available?

Atanu
Databricks Employee
Databricks Employee

@ANOOP V​ , At present, we don’t have these features.

I assume we do plan to include it by Q4 FY2022.

What is next for Multitask Jobs?

After GA we will be working on some highly-requested features during the private preview:

- Job cluster reuse: make it possible to set a job cluster that can be shared across multiple tasks within the same job.

- Task run repair and retry: make it possible to manually restart a DAG execution from a failed step.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group