The best way to write and debug python notebooks

[maikel]

Hello Community,

do you have a suggestion, what is the best way to develop python notebooks in databricks? 
I know that it can be done directly in databricks UI but for more complex projects where I would like to split notebook and import functions from different files it is becoming much more harder to debug. 

The best would be to have a possibility to write notebooks from IDE (I use PyCharm) and have connection with databricks to run it. Is there any convenient way other than plugin databricks from Jetbrains which is rather slow?
I also work with asset bundles.

Thanks a lot in advance for the response!

[stbjelcevic · Accepted solution]

Hi @maikel ,

The PyCharm plugin is generally recommended if you're using PyCharm for local development. I'm curious about the slowness you are reporting, things run slower from the IDE than if you were to run them in Databricks directly? If you're open to trying a different IDE, we generally recommend the Databricks Extension for VS Code for the best local development experience.

There's a table here with recommendations on your dev setup depending on what you are looking for: https://docs.databricks.com/aws/en/dev-tools#which-developer-tool-should-i-use

[pradeep_singh]

Databricks connect V2 lets you simplify local devleopment and testing . 

Couple of links for more information - 

https://www.databricks.com/blog/2023/04/18/use-databricks-anywhere-databricks-connect-v2.html

https://www.youtube.com/watch?v=o4qMWHgT1zM

[saurabh18cs]

Hi @maikel for us databricks extension on vs code to deploy workflow using asset bundles works like a charm. if plugin is not helping out then i suggest you to install databricks CLI separately or point to the one installed together with databricks extension under your environemt variables.

then use cli to deploy your workflows e.g. 

databricks bundle deploy --target test-job

Br