Hello, everyone. I want to ask if there is a way to connect Databricks cluster with SSH interpreter in your IDE? I know about databricks connect but I want to execute the entire code in the cluster.

[BorislavBlagoev]

I want the entire code to be executed on the cluster (Python, SQL, Spark). I'm familiar with Databricks Connect!

[Hubert-Dudek]

I have a great news for you. Databricks tunnel is coming soon and it will be available as extension for Pycharm and Visual Studio Code (and then rest of IDE)

[BorislavBlagoev]

Can you send me link for that? I mean where you have found information about that?

[Hubert-Dudek]

It was on roadmap meeting last week. I haven't found yet information on the internet. Probably some preview or beta is available. You can always contact databricks and ask for access.

[BorislavBlagoev]

This looks great! Thank you!

[ypm]

Awesome, we need this!

[Hubert-Dudek]

I would go with such a approach (but you need to write small script for your IDE):

• you commit to some brunch in git (like staging)
• your IDE executes databricks cli command "databricks repos update" which will perform pull
• your IDE executes databricks cli job command to run notebook from repo

Databricks cli can be executed as a Rest Api, bash/cmd or can be imported as sdk to programming language

[florent]

any news about databricks tunnel?

[Kaniz · Accepted solution]

Hi @Florent POUSSEROT​ and @Borislav Blagoev​ , In early 2022 databricks-tunnel will be available, which will run your code on databricks cloud and not directly on the cluster. There will be ready extensions for PyCharm and VS Code.

[ypm]

HI @Kaniz Fatma​ 

Sorry for asking again, but we are very much looking forward to use this feature. Any news regarding this?

[User16764241763]

@Borislav Blagoev​ 

Could you please give this a try? You can pretty much execute all commands with this integration

https://databricks.com/blog/2019/12/03/jupyterlab-databricks-integration-bridge-local-and-remote-wor...

[BorislavBlagoev]

@Arvind Ravish​  I'm using PyCharm

[Kaniz]

Hi @Borislav Blagoev​ , Please try these:-

• https://medium.com/@manju.sharan797/integrating-databricks-with-pycharm-ide-968d273d48d4
• https://docs.databricks.com/dev-tools/databricks-connect.html#pycharm

[BorislavBlagoev]

@Kaniz Fatma​  I know about the databricks-connect but this feature is not what i want. I'm waiting for databricks-tunnel

[Kaniz]

Hi @Borislav Blagoev​ , Thank you for the update!