Just a doubt, can we develop a kedro project in databricks?

[riccostamendes]

I am asking this because up to now I have just seen some examples of deploying a pre-existent kedro project in databricks in order to run some pipelines...

[LandanG]

Hi @Ricardo Costa Mendes​,

You can integrate Kedro and Databricks together (including running Kedro in Databricks), and the docs are a good place to start, however, I've been seeing companies opt for MLflow 2.0 (recipes) and DLT which essentially does the same thing as Kedro with native support.

[riccostamendes]

yes, you can deploy a pre-existent kedro project in databricks, but as far as I know you cannot create it. you have to create it somewhere else and then deploy it in db.

[noklam]

Hi! Kedro Dev here. You can surely develop Kedro on Databricks, in fact we have a lot of Kedro project running on Databricks. 

In the past there has been some friction, mainly because Kedro are project based while Databricks focus a lot on notebook. This has changed since the introduction of dbx, data assets bundle which allows user work in a IDE while sending job to databricks. There is now a new kedro-databricks community plugin which helps you to get started quickly with data assets bundle https://github.com/JenspederM/kedro-databricks.

> I've been seeing companies opt for MLflow 2.0 (recipes) and DLT which essentially does the same thing as Kedro with native support.

Kedro are quite different from these tools, but I agree it does not make too much sense if you are using Recipe and Kedro at the same time. I won't be able to give a fair comparison since I am obviously biased.