topic Databricks Workflow Orchestration for Pipeline in Data Engineering

Databricks Workflow Orchestration for Pipeline

Pratikmsbsvm — Thu, 31 Jul 2025 04:38:45 GMT

Hello,

I am using Databricks first time. May someone please help me how to do orchestration for the pipeline shown below.

Kindly share the steps how to implement Orchestration , what all steps we have to consider.

Thanks a lot

Re: Databricks Workflow Orchestration for Pipeline

radothede — Thu, 31 Jul 2025 14:09:38 GMT

Hi @Pratikmsbsvm

Please provide some more information and requirements, describe Your scenario - basically what do You want to achive and what do You need.

In general, You most propably want to use jobs / workflows. You can orchestrate your job using schedule or event (file arrival/ table operation) triggers.

Within the job, you want to create tasks that will process your data from source to target, let's say bronze to silver.

In a task logic You want to use some python/ pyspark/ sql code to read data from source, apply transformations (if needed) and write data to target.

For CICD You want to use Databricks Asset Bundles - thats how you want to create your resources in databricks, using Service Principal. Use it with Azure Devops/ Github Actions or any other tool.

Best,

Radek.

Re: Databricks Workflow Orchestration for Pipeline

nayan_wylde — Thu, 31 Jul 2025 15:17:07 GMT

Can you be more specific about the requirement. The diagram is an architecture diagram and it doesn't give the idea about the full requirement. If you are looking ideas to set up a job in databricks. Here is the doc.

https://docs.databricks.com/aws/en/jobs/

Re: Databricks Workflow Orchestration for Pipeline

junaid-databrix — Thu, 31 Jul 2025 17:49:00 GMT

The diagram you have shared is bit confusing: From Azure there is a data pull to Bronze layer, and from the same data source data is being pulled into Silver layer. However, following the Medallion architecture typically the raw data is ingested into Bronze layer, some transformations are applied on Bronze layer data and then it is moved to Silver layer.

BR,

Junaid