- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-26-2022 11:29 PM
For a batch job I can use ADF and Databricks notebook activity to create a pipeline.
Similarly what Azure stack I should use to run Structured streaming Databricks notebook for a production ready pipeline.
- Labels:
-
Databricks notebook
-
Pipeline
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-27-2022 01:29 AM
Hi @Abhijeet Singh I believe you need to ask this query in Azure/Microsoft forum.
@Hubert Dudek / @Werner Stinckens do you have any inputs on this?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-27-2022 01:44 AM
I would run from databricks not ADF but you can start the notebook with the streaming job from ADF, but you need to think about logic, and also, ADF will wait idly for the streaming to finish so that you will pay double. There are many Azure stream options, but you run streaming notebook only from Azure Databricks.
My blog: https://databrickster.medium.com/
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-27-2022 01:44 AM
ok Sure
