Efficiently orchestrate data bricks jobs

[Phani1]

Hi Team,

How efficiently can orchestrate data bricks jobs which involve a lot of transformations, dependencies, and complexity?

At source have a lot of SSIS packages that have complex dependencies and more transformation.     

We have the following options.

1) implement the logic in data bricks notebooks and schedule by using data bricks jobs, workflows

2) Implement the logic in DBT and schedule in DBT 

3) Implement the logic in data bricks notebooks and schedule by using ADF

Could you please suggest what is the best way to implement which has more feasible to re-run and cost-saving? Please share if there is any reference docs/links.

[Anonymous]

Hi @Janga Reddy​ 

Hope everything is going great.

Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we can help you. 

Cheers!

[Phani1]

My question is, how do we reliably orchestrate multiple Databricks Jobs/Workflows that are running in a mixed latency and can write to the same silver and gold delta tables? Could you please suggest the best approach and practices for the same?