Azure Databricks Capabilities: My objective is to evaluate Azure Databricks capability and Do I need to use Azure Devops or Jenkins or Databricks suffice the need.

[Saurabh707344]

hi,

We have Real time streaming usecase where we have to build pipeline using Azure Databricks.

My objective is to evaluate Azure Databricks capability and Do I need to use Azure Devops or Jenkins or Databricks suffice the need.

Can you please provide clarity around few of the below questions around Databricks MLOps :-

Q1. In case i use GIT repos instead of Databricks repos, does it create any conflict when I have usecase of realtime streaming in production ?

In other words Does external tools like GIT provide seamless experience when we have realtime streaming scenarios ?

Q2. Jenkins Or Azure devops service need to be used for build and rest part of automated CI/CD pipelines OR

Does Databricks suffice need of automated pipeline of code ?

Thanks,

Saurabh

[-werners-]

Q1: Databricks repos uses Git (or Azure devops or Gitlab etc) so Databricks repos is just an umbrella term.

If it will work with streaming? Yes. The functionalities of Repos is limited (read: code versioning using branches). If you need prehooks f.e., that is not possible.

You do not have to use Repos, you can work without any git provider as Databricks keeps code (notebook) history itself, although I highly recommend using Repos.

Q2: you can use Azure Devops or Jenkins but it is not a requirement. If you have a small/simple environment, CI can be overkill. But if you need to use automated pipelines, yes that is possible.

[Saurabh707344]

Do you have link how to create automated CI and CD pipelines in Databricks itself, without using any Azure deops or Git+Jenkins ??

[-werners-]

I probably was not clear but you cannot do CI/CD in databricks without an external provider like Jenkins etc.

You can only opt to NOT do it.

[StephanieRivera]

I found these youtube videos to be beneficial.

CI/CD with Azure Dev Ops

Terraform Enablement - Part 1 of 2