Streaming pipeline orchestration

[Abhijeet]

For a batch job I can use ADF and Databricks notebook activity to create a pipeline.

Similarly what Azure stack I should use to run Structured streaming Databricks notebook for a production ready pipeline.

[Prabakar]

Hi @Abhijeet Singh​ I believe you need to ask this query in Azure/Microsoft forum.

@Hubert Dudek​ / @Werner Stinckens​  do you have any inputs on this?

[Hubert-Dudek · Accepted solution]

I would run from databricks not ADF but you can start the notebook with the streaming job from ADF, but you need to think about logic, and also, ADF will wait idly for the streaming to finish so that you will pay double. There are many Azure stream options, but you run streaming notebook only from Azure Databricks.

[Abhijeet]

ok Sure

[Kaniz]

Hi @Abhijeet Singh​ ​, Please don't forget to click on the "Select As Best" button whenever the information helps resolve your question.