Connect to databricks from external non-spark cluster

[Ishmael]

Hi,

I have an app/service on a non-spark kubernetes cluster. Is there a way to access/query a databricks service from my app/service? I see documentations on connectors, particularly on scala which is the code of my app/service. Can I use these connectors from outside the databricks cluster?

[Kaniz]

Hi @Ishmael , You can use Databricks Connect to connect to a Databricks workspace from your local machine or a non...23. Databricks Connect is a client library that allows you to run Spark code from your local machine or a non-Databricks environment such as a Kubernetes cluster. You can use Databricks Connect to connect to a Databricks workspace and run Spark code on a Databric....

[Ishmael]

Thanks Kaniz. I am checking this out. Is this right that it only currently supports scala 2.12? I'm using Scala 2.13 so this might be a problem. Is there an alternative?

[Kaniz]

Hi @Ishmael, 

According to a Databricks community post, Databricks Runtime does not yet support Scala 2.13. However, engineering is targeting the DBR 14 release for Scala 2.13. There is no tentative date for the DBR 14 release yet, but there is usually a DBR version every 6 mo...¹.

In the meantime, you may want to consider using an alternative to Databricks Connect for Scala 2.13. One possible alternative is to use the databricks-connect package from PyPI². The databricks-connect package provides an alternative way to connect to Databricks clusters from your local machine. You can use the package to initialize a connection based on a Config object from the Databricks SDK. The DatabricksSession.builder.sdkConfig(config).getOrCreate() method can be used to create a session...².

I hope this helps!

[Ishmael]

Kaniz do you mean using this python library in scala? I did a quick google search and it seems this is possible. Is this what you mean? It seems ScalaPy is a library that allows this. Is this the library that you recommend? Thanks again.

[Kaniz · Accepted solution]

Hi @Ishmael, Yes, you are correct. ScalaPy is a library that enables seamless interoperability between Scala and Python. It is based on the Py4J project, which allows Python programs running in a Python interpreter to dynamically access Java objects in a Java Virtual Machine. ScalaPy is designed to be easy to use and integrate with existing Scala and Python code. You can find more information and examples on how to use ScalaPy on its [official website]¹ or its [GitHub repository]². I hope this helps! 😊.

[Ishmael]

Thank you very much @Kaniz . I'll try this out.

[Kaniz]

Sure!!! Please come back and let us know how it helped you.