Hello, everyone. I want to ask if there is a way to connect Databricks cluster with SSH interpreter in your IDE? I know about databricks connect but I want to execute the entire code in the cluster.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ12-01-2021 12:51 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ12-07-2021 12:37 AM
I want the entire code to be executed on the cluster (Python, SQL, Spark). I'm familiar with Databricks Connect!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ12-12-2021 08:39 AM
I have a great news for you. Databricks tunnel is coming soon and it will be available as extension for Pycharm and Visual Studio Code (and then rest of IDE)
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ12-13-2021 02:08 AM
Can you send me link for that? I mean where you have found information about that?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ12-13-2021 04:04 AM
It was on roadmap meeting last week. I haven't found yet information on the internet. Probably some preview or beta is available. You can always contact databricks and ask for access.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ12-13-2021 04:39 AM
This looks great! Thank you!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ04-20-2022 01:08 PM
Awesome, we need this!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ12-02-2021 02:46 AM
I would go with such a approach (but you need to write small script for your IDE):
- you commit to some brunch in git (like staging)
- your IDE executes databricks cli command "databricks repos update" which will perform pull
- your IDE executes databricks cli job command to run notebook from repo
Databricks cli can be executed as a Rest Api, bash/cmd or can be imported as sdk to programming language
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ02-02-2022 12:42 PM
any news about databricks tunnel?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ04-20-2022 01:05 PM
HI @Kaniz Fatmaโ
Sorry for asking again, but we are very much looking forward to use this feature. Any news regarding this?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ02-15-2022 06:11 AM
@Borislav Blagoevโ
Could you please give this a try? You can pretty much execute all commands with this integration
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ02-21-2022 07:48 AM
@Arvind Ravishโ I'm using PyCharm
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ02-22-2022 02:30 AM
@Kaniz Fatmaโ I know about the databricks-connect but this feature is not what i want. I'm waiting for databricks-tunnel
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ07-29-2022 11:38 PM
Nice and amazing post this one is, thanks for sharingโฆ
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ08-02-2022 10:30 PM
Thanks for sharing, I found a lot of interesting information here. A really good post, very thankful and helpful that you will write many more posts like this one.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ11-02-2022 01:23 AM
For those of you who are interested, we created an extension for VSCode to work with Databricks which allows you to manage your Databricks workspace (DBFS, Clusters, Secrets, Repos, ...) and also run notebooks interactively against your Databricks Cluster
https://marketplace.visualstudio.com/items?itemName=paiqo.databricks-vscode
It does not require any additional installation (no DBX, no SSH, no specific cluster config, ...) - just your Databricks URL and a Personal Access Token (PAT)