Azure Databricks Capabilities:
My objective is to evaluate Azure Databricks capability and Do I need to use Azure Devops or Jenkins or Databricks suffice the need.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-17-2023 04:26 AM
hi,
We have Real time streaming usecase where we have to build pipeline using Azure Databricks.
My objective is to evaluate Azure Databricks capability and Do I need to use Azure Devops or Jenkins or Databricks suffice the need.
Can you please provide clarity around few of the below questions around Databricks MLOps :-
Q1. In case i use GIT repos instead of Databricks repos, does it create any conflict when I have usecase of realtime streaming in production ?
In other words Does external tools like GIT provide seamless experience when we have realtime streaming scenarios ?
Q2. Jenkins Or Azure devops service need to be used for build and rest part of automated CI/CD pipelines OR
Does Databricks suffice need of automated pipeline of code ?
Thanks,
Saurabh
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-25-2023 03:07 AM
Q1: Databricks repos uses Git (or Azure devops or Gitlab etc) so Databricks repos is just an umbrella term.
If it will work with streaming? Yes. The functionalities of Repos is limited (read: code versioning using branches). If you need prehooks f.e., that is not possible.
You do not have to use Repos, you can work without any git provider as Databricks keeps code (notebook) history itself, although I highly recommend using Repos.
Q2: you can use Azure Devops or Jenkins but it is not a requirement. If you have a small/simple environment, CI can be overkill. But if you need to use automated pipelines, yes that is possible.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-25-2023 03:59 AM
Do you have link how to create automated CI and CD pipelines in Databricks itself, without using any Azure deops or Git+Jenkins ??
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-25-2023 04:06 AM
I probably was not clear but you cannot do CI/CD in databricks without an external provider like Jenkins etc.
You can only opt to NOT do it.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-31-2023 10:53 AM
I found these youtube videos to be beneficial.