DatabricksWorkflowTaskGroup

Nik_Vanderhoof
Contributor

Hello,

I recently learned about the DatabricksWorkflowTaskGroup operator for Airflow that allows one to run multiple Notebook tasks on a shared job compute cluster from Airflow.

Is a similar feature possible to run multiple non-Notebook tasks from Airflow on a shared job cluster? That would be of great use to us.

Thanks!

Alberto_Umana
Databricks Employee
Databricks Employee

Hi @Nik_Vanderhoof,

Yes, it is possible using DatabricksWorkflowTaskGroup. You can include the DatabricksTaskOperator for non-notebook tasks.

 

 

View solution in original post

Nik_Vanderhoof
Contributor

Thank you!