Hello, everyone. I want to ask if there is a way to connect Databricks cluster with SSH interpreter in your IDE? I know about databricks connect but I want to execute the entire code in the cluster.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-01-2021 12:51 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-02-2021 02:00 AM
Hi @Borislav Blagoev unfortunately it is not possible to connect to the cluster other Databricks connect.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-02-2021 02:04 AM
Is it possible to execute the entire code in the databricks cluster instead only the spark code?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-02-2021 02:31 AM
For Spark jobs, you can use Databricks connect.
To use Python code to run SQL commands on Databricks clusters and Databricks SQL endpoints you can use the Databricks SQL Connector for Python.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-02-2021 04:18 AM
I want to execute Python code as well. The entire code (Spark, SQL, Python).
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-02-2021 07:47 AM
hm I think plain python code will run with databricks connect (if it is a python program you are writing), and spark sql can be done by spark.sql(...).
Is that what you want to do?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-02-2021 01:36 PM
Only the spark code is executed in the cluster. Unfortunately!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-03-2021 12:51 AM
dang, not even the spark.sql("...")?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-03-2021 01:35 AM
As I mentioned earlier, only spark codes will be executed with Databricks connect. We have an internal feature request to access the Python REPL from the local IDE through DBconnect.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-03-2021 02:48 AM
I don't know why but when I want to access that link I get this error: Unable to sign in I tried with the same email as here.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-03-2021 02:54 AM
@Werner Stinckens you can execute spark.sql("...") in the cluster but I want to execute this for example:
collection = [1, 2, 3, 4, 5]
sum = 0
for x in collection:
sum += x
stupid example!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-03-2021 03:04 AM
Hi @Borislav Blagoev , you won't be able to access it. As I mentioned in the previous comment it's an internal feature request and only available for Databricks employees.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-03-2021 04:54 AM
Oh, OK! I didn't understand that sorry!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-06-2021 04:32 PM
hi @Borislav Blagoev
Have you check the list of limitation for DB connect? docs here https://docs.databricks.com/dev-tools/databricks-connect.html#limitations
Limitations
The following Databricks features and third-party platforms are unsupported:
- Structured Streaming.
- Running arbitrary code that is not a part of a Spark job on the remote cluster.
- Native Scala, Python, and R APIs for Delta table operations (for example, DeltaTable.forPath) are not supported. However, the SQL API (spark.sql(...)) with Delta Lake operations and the Spark API (for example, spark.read.load) on Delta tables are both supported.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-07-2021 12:36 AM
Yes, that's why I want to use something different than Databrcks Connect!

