topic Re: Just a doubt, can we develop a kedro project in databricks? in Data Engineering

Just a doubt, can we develop a kedro project in databricks?

riccostamendes — Thu, 26 Jan 2023 01:21:22 GMT

I am asking this because up to now I have just seen some examples of deploying a pre-existent kedro project in databricks in order to run some pipelines...

Re: Just a doubt, can we develop a kedro project in databricks?

LandanG — Thu, 26 Jan 2023 14:23:49 GMT

Hi @Ricardo Costa Mendes,

You can integrate Kedro and Databricks together (including running Kedro in Databricks), and the docs are a good place to start, however, I've been seeing companies opt for MLflow 2.0 (recipes) and DLT which essentially does the same thing as Kedro with native support.

Re: Just a doubt, can we develop a kedro project in databricks?

riccostamendes — Thu, 26 Jan 2023 15:19:10 GMT

yes, you can deploy a pre-existent kedro project in databricks, but as far as I know you cannot create it. you have to create it somewhere else and then deploy it in db.

Re: Just a doubt, can we develop a kedro project in databricks?

noklam — Fri, 02 Aug 2024 16:40:15 GMT

Hi! Kedro Dev here. You can surely develop Kedro on Databricks, in fact we have a lot of Kedro project running on Databricks.

In the past there has been some friction, mainly because Kedro are project based while Databricks focus a lot on notebook. This has changed since the introduction of dbx, data assets bundle which allows user work in a IDE while sending job to databricks. There is now a new kedro-databricks community plugin which helps you to get started quickly with data assets bundle https://github.com/JenspederM/kedro-databricks.

> I've been seeing companies opt for MLflow 2.0 (recipes) and DLT which essentially does the same thing as Kedro with native support.

Kedro are quite different from these tools, but I agree it does not make too much sense if you are using Recipe and Kedro at the same time. I won't be able to give a fair comparison since I am obviously biased.