topic How do I import class/functions so it work in Databricks and in my IDE in Data Engineering

How do I import class/functions so it work in Databricks and in my IDE

avnerrhh — Thu, 25 Nov 2021 13:31:29 GMT

I already saw this post

I want my code to work on both platforms (Databricks and PyCharm), is there any way to do it?

Re: How do I import class/functions so it work in Databricks and in my IDE

-werners- — Thu, 25 Nov 2021 13:37:38 GMT

yes.

one way is to develop everything locally on your pc, so you also need to have spark installed.

This is of course not ideal as you will not have some interesting stuff that databricks provides.

But it can be done. What you have to do is create a whl and put that on databricks.

A much better way imo is to use databricks-connect. This tool enables you to use your IDE (pycharm) and let the code run on databricks, not locally:

https://docs.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect

Mind that the databricks-connect devs don't follow the ultrafast release schedule of the databricks platform.

Since a short while you can also use Databricks repos for modules and functions:

https://docs.microsoft.com/en-us/azure/databricks/repos#work-with-python-and-r-modules

Re: How do I import class/functions so it work in Databricks and in my IDE

avnerrhh — Thu, 25 Nov 2021 13:53:57 GMT

Yes I've used Databricks-connect to do so.

When I work with pychram '%run' doesn't work but 'import' do, when I work with Databricks 'import' doesn't but '%run' do.

How can I resolve this?

Re: How do I import class/functions so it work in Databricks and in my IDE

avnerrhh — Thu, 25 Nov 2021 13:56:05 GMT

Never mind I found that I need to use 'Files' to do so, Thanks 😄

Re: How do I import class/functions so it work in Databricks and in my IDE

Bruno_Lannoo — Mon, 19 Sep 2022 08:00:20 GMT

Hi @Avner Huri ,

Would you mind elaborating on what you mean by using 'Files'?

It seems I have the same challenge:

1) I already use databricks-connect

2) I still have an issue exploring some notebooks in Pycharm because %run magic is not treated as an import

Re: How do I import class/functions so it work in Databricks and in my IDE

avnerrhh — Mon, 19 Sep 2022 08:14:42 GMT

Files - if you want to 'import' command to Import code to your notebook, your code must be in a 'File'(this is his name on the Databricks UI) if you want to import a notebook you have to use %run

yes the %run command is a problem, I didn't try to solve it just didn't use it in places that must have it.

but I thunk you can just create a wrapper class that use %run when you are on Databricks and import otherwise.

Re: How do I import class/functions so it work in Databricks and in my IDE

Bruno_Lannoo — Mon, 19 Sep 2022 17:08:51 GMT

Oh k, thanks.

That wrapper idea is interesting, maybe I could do something like that.