Hi Team,
How efficiently can orchestrate data bricks jobs which involve a lot of transformations, dependencies, and complexity?
At source have a lot of SSIS packages that have complex dependencies and more transformation.
We have the following options.
1) implement the logic in data bricks notebooks and schedule by using data bricks jobs, workflows
2) Implement the logic in DBT and schedule in DBT
3) Implement the logic in data bricks notebooks and schedule by using ADF
Could you please suggest what is the best way to implement which has more feasible to re-run and cost-saving? Please share if there is any reference docs/links.