Just a doubt, can we develop a kedro project in databricks?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-25-2023 05:21 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-26-2023 06:23 AM
Hi @Ricardo Costa Mendes,
You can integrate Kedro and Databricks together (including running Kedro in Databricks), and the docs are a good place to start, however, I've been seeing companies opt for MLflow 2.0 (recipes) and DLT which essentially does the same thing as Kedro with native support.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-26-2023 07:19 AM
yes, you can deploy a pre-existent kedro project in databricks, but as far as I know you cannot create it. you have to create it somewhere else and then deploy it in db.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-02-2024 09:40 AM
Hi! Kedro Dev here. You can surely develop Kedro on Databricks, in fact we have a lot of Kedro project running on Databricks.
In the past there has been some friction, mainly because Kedro are project based while Databricks focus a lot on notebook. This has changed since the introduction of dbx, data assets bundle which allows user work in a IDE while sending job to databricks. There is now a new kedro-databricks community plugin which helps you to get started quickly with data assets bundle https://github.com/JenspederM/kedro-databricks.
> I've been seeing companies opt for MLflow 2.0 (recipes) and DLT which essentially does the same thing as Kedro with native support.
Kedro are quite different from these tools, but I agree it does not make too much sense if you are using Recipe and Kedro at the same time. I won't be able to give a fair comparison since I am obviously biased.
![](/skins/images/B38AF44D4BD6CE643D2A527BE673CCF6/responsive_peak/images/icon_anonymous_message.png)
![](/skins/images/B38AF44D4BD6CE643D2A527BE673CCF6/responsive_peak/images/icon_anonymous_message.png)