DatabricksWorkflowTaskGroup

[Nik_Vanderhoof]

Hello,

I recently learned about the DatabricksWorkflowTaskGroup operator for Airflow that allows one to run multiple Notebook tasks on a shared job compute cluster from Airflow.

Is a similar feature possible to run multiple non-Notebook tasks from Airflow on a shared job cluster? That would be of great use to us.

Thanks!

[Alberto_Umana · Accepted solution]

Hi @Nik_Vanderhoof,

Yes, it is possible using DatabricksWorkflowTaskGroup. You can include the DatabricksTaskOperator for non-notebook tasks.

[Nik_Vanderhoof]

Thank you!