topic Re: Efficiently orchestrate data bricks jobs in Data Engineering

Efficiently orchestrate data bricks jobs

Phani1 — Thu, 16 Mar 2023 13:53:07 GMT

Hi Team,

How efficiently can orchestrate data bricks jobs which involve a lot of transformations, dependencies, and complexity?

At source have a lot of SSIS packages that have complex dependencies and more transformation.

We have the following options.

1) implement the logic in data bricks notebooks and schedule by using data bricks jobs, workflows

2) Implement the logic in DBT and schedule in DBT

3) Implement the logic in data bricks notebooks and schedule by using ADF

Could you please suggest what is the best way to implement which has more feasible to re-run and cost-saving? Please share if there is any reference docs/links.

Re: Efficiently orchestrate data bricks jobs

Anonymous — Mon, 20 Mar 2023 04:02:29 GMT

Hi @Janga Reddy

Hope everything is going great.

Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we can help you.

Cheers!

Re: Efficiently orchestrate data bricks jobs

Phani1 — Tue, 21 Mar 2023 04:31:05 GMT

My question is, how do we reliably orchestrate multiple Databricks Jobs/Workflows that are running in a mixed latency and can write to the same silver and gold delta tables? Could you please suggest the best approach and practices for the same?