Complex Embedded Workflows

sandeepmankikar — Wed, 12 Mar 2025 04:31:08 GMT

Can complex embedded workflows be created using Databricks Bundle, where multiple workflows are interconnected in a parent-child format? If Databricks Bundle doesn't support this, what would be the best alternative for creating and deploying such workflows across different environments?

Re: Complex Embedded Workflows

ashraf1395 — Wed, 12 Mar 2025 04:50:35 GMT

Yup you can create , complex workflows as well in databricks bundles , some examples can be
You can have all of them defined from child to parent and can call those chirl workflows as workflow task in the parent workflows, all different kinds of tasks like if-else tasks, foreach task, notebook task ,dlt pipeline task , SQL task, workflow task etc can be defined. You can create separate yaml files for each of those workflows as well by following standard naming conventions or if you want you can have all of them in a single bundle.resources.yml file as well.

here are some links which can be helpful
- https://docs.databricks.com/aws/en/dev-tools/bundles/resources
- https://docs.databricks.com/aws/en/dev-tools/bundles/resource-examples
- https://docs.databricks.com/aws/en/dev-tools/bundles/job-task-types
- https://docs.databricks.com/aws/en/dev-tools/bundles/job-task-override

I guess these might be helpful.

topic Complex Embedded Workflows in Data Engineering

Complex Embedded Workflows

Re: Complex Embedded Workflows