Databricks Workflow Orchestration for Pipeline

[Pratikmsbsvm]

Hello,

I am using Databricks first time. May someone please help me how to do orchestration for the pipeline shown below.

Kindly share the steps how to implement Orchestration , what all steps we have to consider.

Thanks a lot

[radothede]

Hi @Pratikmsbsvm 

Please provide some more information and requirements, describe Your scenario - basically what do You want to achive and what do You need.

In general, You most propably want to use jobs / workflows. You can orchestrate your job using schedule or event (file arrival/ table operation) triggers. 

Within the job, you want to create tasks that will process your data from source to target, let's say bronze to silver.

In a task logic You want to use some python/ pyspark/ sql code to read data from source, apply transformations (if needed) and write data to target.

For CICD You want to use Databricks Asset Bundles - thats how you want to create your resources in databricks, using Service Principal. Use it with Azure Devops/ Github Actions or any other tool.

Best,

Radek.

[nayan_wylde]

Can you be more specific about the requirement. The diagram is an architecture diagram and it doesn't give the idea about the full requirement. If you are looking ideas to set up a job in databricks. Here is the doc.

https://docs.databricks.com/aws/en/jobs/

[junaid-databrix]

The diagram you have shared is bit confusing: From Azure there is a data pull to Bronze layer, and from the same data source data is being pulled into Silver layer. However, following the Medallion architecture typically the raw data is ingested into Bronze layer, some transformations are applied on Bronze layer data and then it is moved to Silver layer.

BR,

Junaid