Hello, everyone. I want to ask if there is a way to connect Databricks cluster with SSH interpreter in your IDE? I know about databricks connect but I want to execute the entire code in the cluster.

[BorislavBlagoev]

I want the entire code to be executed on the cluster (Python, SQL, Spark). I'm familiar with Databricks Connect!

[Hubert-Dudek]

I have a great news for you. Databricks tunnel is coming soon and it will be available as extension for Pycharm and Visual Studio Code (and then rest of IDE)

[BorislavBlagoev]

Can you send me link for that? I mean where you have found information about that?

[Hubert-Dudek]

It was on roadmap meeting last week. I haven't found yet information on the internet. Probably some preview or beta is available. You can always contact databricks and ask for access.

[BorislavBlagoev]

This looks great! Thank you!

[ypm]

Awesome, we need this!

[Hubert-Dudek]

I would go with such a approach (but you need to write small script for your IDE):

• you commit to some brunch in git (like staging)
• your IDE executes databricks cli command "databricks repos update" which will perform pull
• your IDE executes databricks cli job command to run notebook from repo

Databricks cli can be executed as a Rest Api, bash/cmd or can be imported as sdk to programming language

[florent]

any news about databricks tunnel?

[ypm]

HI @Kaniz Fatma​ 

Sorry for asking again, but we are very much looking forward to use this feature. Any news regarding this?

[User16764241763]

@Borislav Blagoev​ 

Could you please give this a try? You can pretty much execute all commands with this integration

https://databricks.com/blog/2019/12/03/jupyterlab-databricks-integration-bridge-local-and-remote-wor...

[BorislavBlagoev]

@Arvind Ravish​  I'm using PyCharm

[BorislavBlagoev]

@Kaniz Fatma​  I know about the databricks-connect but this feature is not what i want. I'm waiting for databricks-tunnel

[Anonymous]

Nice and amazing post this one is, thanks for sharing…

Subway Live IQ

[Anonymous]

Thanks for sharing, I found a lot of interesting information here. A really good post, very thankful and helpful that you will write many more posts like this one.

MyMileStoneCard

[gbrueckl]

For those of you who are interested, we created an extension for VSCode to work with Databricks which allows you to manage your Databricks workspace (DBFS, Clusters, Secrets, Repos, ...) and also run notebooks interactively against your Databricks Cluster

https://marketplace.visualstudio.com/items?itemName=paiqo.databricks-vscode

It does not require any additional installation (no DBX, no SSH, no specific cluster config, ...) - just your Databricks URL and a Personal Access Token (PAT)