topic Re: Hello, everyone. I want to ask if there is a way to connect Databricks cluster with SSH interpreter in your IDE? I know about databricks connect but I want to execute the entire code in the cluster. in Data Engineering

Hello, everyone. I want to ask if there is a way to connect Databricks cluster with SSH interpreter in your IDE? I know about databricks connect but I want to execute the entire code in the cluster.

BorislavBlagoev — Wed, 01 Dec 2021 20:51:01 GMT

Re: Hello, everyone. I want to ask if there is a way to connect Databricks cluster with SSH interpreter in your IDE? I know about databricks connect but I want to execute the entire code in the cluster.

Prabakar — Thu, 02 Dec 2021 10:00:17 GMT

Hi @Borislav Blagoev unfortunately it is not possible to connect to the cluster other Databricks connect.

Re: Hello, everyone. I want to ask if there is a way to connect Databricks cluster with SSH interpreter in your IDE? I know about databricks connect but I want to execute the entire code in the cluster.

BorislavBlagoev — Thu, 02 Dec 2021 10:04:43 GMT

Is it possible to execute the entire code in the databricks cluster instead only the spark code?

Re: Hello, everyone. I want to ask if there is a way to connect Databricks cluster with SSH interpreter in your IDE? I know about databricks connect but I want to execute the entire code in the cluster.

Prabakar — Thu, 02 Dec 2021 10:31:48 GMT

For Spark jobs, you can use Databricks connect.

To use Python code to run SQL commands on Databricks clusters and Databricks SQL endpoints you can use the Databricks SQL Connector for Python.

Re: Hello, everyone. I want to ask if there is a way to connect Databricks cluster with SSH interpreter in your IDE? I know about databricks connect but I want to execute the entire code in the cluster.

Hubert-Dudek — Thu, 02 Dec 2021 10:46:18 GMT

I would go with such a approach (but you need to write small script for your IDE):

you commit to some brunch in git (like staging)
your IDE executes databricks cli command "databricks repos update" which will perform pull
your IDE executes databricks cli job command to run notebook from repo

Databricks cli can be executed as a Rest Api, bash/cmd or can be imported as sdk to programming language

Re: Hello, everyone. I want to ask if there is a way to connect Databricks cluster with SSH interpreter in your IDE? I know about databricks connect but I want to execute the entire code in the cluster.

BorislavBlagoev — Thu, 02 Dec 2021 12:18:53 GMT

I want to execute Python code as well. The entire code (Spark, SQL, Python).

Re: Hello, everyone. I want to ask if there is a way to connect Databricks cluster with SSH interpreter in your IDE? I know about databricks connect but I want to execute the entire code in the cluster.

-werners- — Thu, 02 Dec 2021 15:47:36 GMT

hm I think plain python code will run with databricks connect (if it is a python program you are writing), and spark sql can be done by spark.sql(...).

Is that what you want to do?

Re: Hello, everyone. I want to ask if there is a way to connect Databricks cluster with SSH interpreter in your IDE? I know about databricks connect but I want to execute the entire code in the cluster.

BorislavBlagoev — Thu, 02 Dec 2021 21:36:59 GMT

Only the spark code is executed in the cluster. Unfortunately!

Re: Hello, everyone. I want to ask if there is a way to connect Databricks cluster with SSH interpreter in your IDE? I know about databricks connect but I want to execute the entire code in the cluster.

-werners- — Fri, 03 Dec 2021 08:51:56 GMT

dang, not even the spark.sql("...")?

Re: Hello, everyone. I want to ask if there is a way to connect Databricks cluster with SSH interpreter in your IDE? I know about databricks connect but I want to execute the entire code in the cluster.

Prabakar — Fri, 03 Dec 2021 09:35:16 GMT

As I mentioned earlier, only spark codes will be executed with Databricks connect. We have an internal feature request to access the Python REPL from the local IDE through DBconnect.

Re: Hello, everyone. I want to ask if there is a way to connect Databricks cluster with SSH interpreter in your IDE? I know about databricks connect but I want to execute the entire code in the cluster.

BorislavBlagoev — Fri, 03 Dec 2021 10:48:18 GMT

I don't know why but when I want to access that link I get this error: Unable to sign in I tried with the same email as here.

Re: Hello, everyone. I want to ask if there is a way to connect Databricks cluster with SSH interpreter in your IDE? I know about databricks connect but I want to execute the entire code in the cluster.

BorislavBlagoev — Fri, 03 Dec 2021 10:54:35 GMT

@Werner Stinckens you can execute spark.sql("...") in the cluster but I want to execute this for example:

collection = [1, 2, 3, 4, 5]
sum = 0
for x in collection:
     sum += x

stupid example!

Re: Hello, everyone. I want to ask if there is a way to connect Databricks cluster with SSH interpreter in your IDE? I know about databricks connect but I want to execute the entire code in the cluster.

Prabakar — Fri, 03 Dec 2021 11:04:52 GMT

Hi @Borislav Blagoev , you won't be able to access it. As I mentioned in the previous comment it's an internal feature request and only available for Databricks employees.

Re: Hello, everyone. I want to ask if there is a way to connect Databricks cluster with SSH interpreter in your IDE? I know about databricks connect but I want to execute the entire code in the cluster.

BorislavBlagoev — Fri, 03 Dec 2021 12:54:41 GMT

Oh, OK! I didn't understand that sorry!

Re: Hello, everyone. I want to ask if there is a way to connect Databricks cluster with SSH interpreter in your IDE? I know about databricks connect but I want to execute the entire code in the cluster.

jose_gonzalez — Tue, 07 Dec 2021 00:32:36 GMT

hi @Borislav Blagoev

Have you check the list of limitation for DB connect? docs here https://docs.databricks.com/dev-tools/databricks-connect.html#limitations

Limitations

The following Databricks features and third-party platforms are unsupported:

Structured Streaming.
Running arbitrary code that is not a part of a Spark job on the remote cluster.
Native Scala, Python, and R APIs for Delta table operations (for example, DeltaTable.forPath) are not supported. However, the SQL API (spark.sql(...)) with Delta Lake operations and the Spark API (for example, spark.read.load) on Delta tables are both supported.

Re: Hello, everyone. I want to ask if there is a way to connect Databricks cluster with SSH interpreter in your IDE? I know about databricks connect but I want to execute the entire code in the cluster.

BorislavBlagoev — Tue, 07 Dec 2021 08:36:05 GMT

Yes, that's why I want to use something different than Databrcks Connect!

Re: Hello, everyone. I want to ask if there is a way to connect Databricks cluster with SSH interpreter in your IDE? I know about databricks connect but I want to execute the entire code in the cluster.

BorislavBlagoev — Tue, 07 Dec 2021 08:37:29 GMT

I want the entire code to be executed on the cluster (Python, SQL, Spark). I'm familiar with Databricks Connect!

Re: Hello, everyone. I want to ask if there is a way to connect Databricks cluster with SSH interpreter in your IDE? I know about databricks connect but I want to execute the entire code in the cluster.

Hubert-Dudek — Sun, 12 Dec 2021 16:39:45 GMT

I have a great news for you. Databricks tunnel is coming soon and it will be available as extension for Pycharm and Visual Studio Code (and then rest of IDE)

Re: Hello, everyone. I want to ask if there is a way to connect Databricks cluster with SSH interpreter in your IDE? I know about databricks connect but I want to execute the entire code in the cluster.

BorislavBlagoev — Mon, 13 Dec 2021 10:08:51 GMT

Can you send me link for that? I mean where you have found information about that?

Re: Hello, everyone. I want to ask if there is a way to connect Databricks cluster with SSH interpreter in your IDE? I know about databricks connect but I want to execute the entire code in the cluster.

Hubert-Dudek — Mon, 13 Dec 2021 12:04:59 GMT

It was on roadmap meeting last week. I haven't found yet information on the internet. Probably some preview or beta is available. You can always contact databricks and ask for access.