cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Complex Embedded Workflows

sandeepmankikar
Contributor

Can complex embedded workflows be created using Databricks Bundle, where multiple workflows are interconnected in a parent-child format? If Databricks Bundle doesn't support this, what would be the best alternative for creating and deploying such workflows across different environments?

1 REPLY 1

ashraf1395
Honored Contributor

Yup you can create , complex workflows as well in databricks bundles , some examples can be
You can have all of them defined from child to parent and can call those chirl workflows as workflow task in the parent workflows, all different kinds of tasks like if-else tasks, foreach task, notebook task ,dlt pipeline task , SQL task, workflow task etc can be defined. You can create separate yaml files for each of those workflows as well by following standard naming conventions or if you want you can have all of them in a single bundle.resources.yml file as well.

here are some links which can be helpful
https://docs.databricks.com/aws/en/dev-tools/bundles/resources
https://docs.databricks.com/aws/en/dev-tools/bundles/resource-examples
https://docs.databricks.com/aws/en/dev-tools/bundles/job-task-types
https://docs.databricks.com/aws/en/dev-tools/bundles/job-task-override

I guess these might be helpful.

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local communityโ€”sign up today to get started!

Sign Up Now