cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Just a doubt, can we develop a kedro project in databricks?

riccostamendes
New Contributor II

I am asking this because up to now I have just seen some examples of deploying a pre-existent kedro project in databricks in order to run some pipelines...

3 REPLIES 3

LandanG
Honored Contributor

Hi @Ricardo Costa Mendesโ€‹,

You can integrate Kedro and Databricks together (including running Kedro in Databricks), and the docs are a good place to start, however, I've been seeing companies opt for MLflow 2.0 (recipes) and DLT which essentially does the same thing as Kedro with native support.

riccostamendes
New Contributor II

yes, you can deploy a pre-existent kedro project in databricks, but as far as I know you cannot create it. you have to create it somewhere else and then deploy it in db.

noklam
New Contributor II

Hi! Kedro Dev here. You can surely develop Kedro on Databricks, in fact we have a lot of Kedro project running on Databricks. 

In the past there has been some friction, mainly because Kedro are project based while Databricks focus a lot on notebook. This has changed since the introduction of dbx, data assets bundle which allows user work in a IDE while sending job to databricks. There is now a new kedro-databricks community plugin which helps you to get started quickly with data assets bundle https://github.com/JenspederM/kedro-databricks.

I've been seeing companies opt for MLflow 2.0 (recipes) and DLT which essentially does the same thing as Kedro with native support.

Kedro are quite different from these tools, but I agree it does not make too much sense if you are using Recipe and Kedro at the same time. I won't be able to give a fair comparison since I am obviously biased.

 

 

 

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group