cancel
Showing results for 
Search instead for 
Did you mean: 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results for 
Search instead for 
Did you mean: 

Azure Databricks Capabilities: My objective is to evaluate Azure Databricks capability and Do I need to use Azure Devops or Jenkins or Databricks suffice the need.

Saurabh707344
New Contributor III

hi,

We have Real time streaming usecase where we have to build pipeline using Azure Databricks.

My objective is to evaluate Azure Databricks capability and Do I need to use Azure Devops or Jenkins or Databricks suffice the need.

Can you please provide clarity around few of the below questions around Databricks MLOps :-

Q1. In case i use GIT repos instead of Databricks repos, does it create any conflict when I have usecase of realtime streaming in production ?

In other words Does external tools like GIT provide seamless experience when we have realtime streaming scenarios ?

Q2. Jenkins Or Azure devops service need to be used for build and rest part of automated CI/CD pipelines OR

Does Databricks suffice need of automated pipeline of code ?

Thanks,

Saurabh

4 REPLIES 4

-werners-
Esteemed Contributor III

Q1: Databricks repos uses Git (or Azure devops or Gitlab etc) so Databricks repos is just an umbrella term.

If it will work with streaming? Yes. The functionalities of Repos is limited (read: code versioning using branches). If you need prehooks f.e., that is not possible.

You do not have to use Repos, you can work without any git provider as Databricks keeps code (notebook) history itself, although I highly recommend using Repos.

Q2: you can use Azure Devops or Jenkins but it is not a requirement. If you have a small/simple environment, CI can be overkill. But if you need to use automated pipelines, yes that is possible.

Do you have link how to create automated CI and CD pipelines in Databricks itself, without using any Azure deops or Git+Jenkins ??

-werners-
Esteemed Contributor III

I probably was not clear but you cannot do CI/CD in databricks without an external provider like Jenkins etc.

You can only opt to NOT do it.

StephanieRivera
Valued Contributor II
Valued Contributor II

I found these youtube videos to be beneficial.

CI/CD with Azure Dev Ops

Terraform Enablement - Part 1 of 2

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!