topic Re: Azure Databricks Capabilities: My objective is to evaluate Azure Databricks capability and Do I need to use Azure Devops or Jenkins or Databricks suffice the need. in Machine Learning

Azure Databricks Capabilities: My objective is to evaluate Azure Databricks capability and Do I need to use Azure Devops or Jenkins or Databricks suffice the need.

Saurabh707344 — Wed, 17 May 2023 11:26:58 GMT

hi,

We have Real time streaming usecase where we have to build pipeline using Azure Databricks.

My objective is to evaluate Azure Databricks capability and Do I need to use Azure Devops or Jenkins or Databricks suffice the need.

Can you please provide clarity around few of the below questions around Databricks MLOps :-

Q1. In case i use GIT repos instead of Databricks repos, does it create any conflict when I have usecase of realtime streaming in production ?

In other words Does external tools like GIT provide seamless experience when we have realtime streaming scenarios ?

Q2. Jenkins Or Azure devops service need to be used for build and rest part of automated CI/CD pipelines OR

Does Databricks suffice need of automated pipeline of code ?

Thanks,

Saurabh

Re: Azure Databricks Capabilities: My objective is to evaluate Azure Databricks capability and Do I need to use Azure Devops or Jenkins or Databricks suffice the need.

-werners- — Thu, 25 May 2023 10:07:42 GMT

Q1: Databricks repos uses Git (or Azure devops or Gitlab etc) so Databricks repos is just an umbrella term.

If it will work with streaming? Yes. The functionalities of Repos is limited (read: code versioning using branches). If you need prehooks f.e., that is not possible.

You do not have to use Repos, you can work without any git provider as Databricks keeps code (notebook) history itself, although I highly recommend using Repos.

Q2: you can use Azure Devops or Jenkins but it is not a requirement. If you have a small/simple environment, CI can be overkill. But if you need to use automated pipelines, yes that is possible.

Re: Azure Databricks Capabilities: My objective is to evaluate Azure Databricks capability and Do I need to use Azure Devops or Jenkins or Databricks suffice the need.

Saurabh707344 — Thu, 25 May 2023 10:59:19 GMT

Do you have link how to create automated CI and CD pipelines in Databricks itself, without using any Azure deops or Git+Jenkins ??

Re: Azure Databricks Capabilities: My objective is to evaluate Azure Databricks capability and Do I need to use Azure Devops or Jenkins or Databricks suffice the need.

-werners- — Thu, 25 May 2023 11:06:03 GMT

I probably was not clear but you cannot do CI/CD in databricks without an external provider like Jenkins etc.

You can only opt to NOT do it.

Re: Azure Databricks Capabilities: My objective is to evaluate Azure Databricks capability and Do I need to use Azure Devops or Jenkins or Databricks suffice the need.

StephanieAlba — Wed, 31 May 2023 17:53:05 GMT

I found these youtube videos to be beneficial.

CI/CD with Azure Dev Ops

Terraform Enablement - Part 1 of 2