topic Re: Streaming pipeline orchestration in Data Engineering

Streaming pipeline orchestration

Abhijeet — Tue, 27 Sep 2022 06:29:12 GMT

For a batch job I can use ADF and Databricks notebook activity to create a pipeline.

Similarly what Azure stack I should use to run Structured streaming Databricks notebook for a production ready pipeline.

Re: Streaming pipeline orchestration

Prabakar — Tue, 27 Sep 2022 08:29:48 GMT

Hi @Abhijeet Singh I believe you need to ask this query in Azure/Microsoft forum.

@Hubert Dudek / @Werner Stinckens do you have any inputs on this?

Re: Streaming pipeline orchestration

Hubert-Dudek — Tue, 27 Sep 2022 08:44:42 GMT

I would run from databricks not ADF but you can start the notebook with the streaming job from ADF, but you need to think about logic, and also, ADF will wait idly for the streaming to finish so that you will pay double. There are many Azure stream options, but you run streaming notebook only from Azure Databricks.

Re: Streaming pipeline orchestration

Abhijeet — Tue, 27 Sep 2022 08:44:54 GMT

ok Sure