topic DatabricksWorkflowTaskGroup in Get Started Discussions

DatabricksWorkflowTaskGroup

Nik_Vanderhoof — Fri, 28 Feb 2025 15:08:10 GMT

Hello,

I recently learned about the DatabricksWorkflowTaskGroup operator for Airflow that allows one to run multiple Notebook tasks on a shared job compute cluster from Airflow.

Is a similar feature possible to run multiple non-Notebook tasks from Airflow on a shared job cluster? That would be of great use to us.

Thanks!

Re: DatabricksWorkflowTaskGroup

Alberto_Umana — Sat, 01 Mar 2025 20:29:29 GMT

Hi @Nik_Vanderhoof,

Yes, it is possible using DatabricksWorkflowTaskGroup. You can include the DatabricksTaskOperator for non-notebook tasks.

Re: DatabricksWorkflowTaskGroup

Nik_Vanderhoof — Mon, 03 Mar 2025 14:54:34 GMT

Thank you!