topic Re: Issue connecting to SQL warehouse spark thrift server in Data Engineering

Issue connecting to SQL warehouse spark thrift server

AlexDavies — Fri, 27 Jan 2023 13:05:40 GMT

we have a library that allows dotnet applications to talk to databricks clusters (https://github.com/clearbank/SparkSqlClient). This communicates with the clusters over the spark thrift server

Although this works great for clusters in the "data science and engineering" workspace it cant seem to connect to any of the sql warehouse clusters

Do the sql warehouse clusters use a different spark sql thrift server than the other clusters?

Where can I get a copy of the thrift file that describes the endpoint that sql warehouses endpoints support?

Re: Issue connecting to SQL warehouse spark thrift server

Anonymous — Sat, 28 Jan 2023 14:03:53 GMT

@Alex Davies It should be the same thrift server for SQL warehouse.

You can find the connection URLs in connection details tab of the SQL warehouse. Also Databricks provides native connector for multiple partner tools. Please refer here

Re: Issue connecting to SQL warehouse spark thrift server

AlexDavies — Mon, 30 Jan 2023 09:08:18 GMT

I have tried those connection details however it they give me 400 errors when trying to connect directly using the hive thrift server contract (https://github.com/apache/hive/blob/master/service-rpc/if/TCLIService.thrift). I do not get the issues when connecting to data science and engineering clusters. Do the sql warehouse use a slightly different contract? Could I get access to the thrift file the sql warehouses use?

I know there are lots of connectors, however there are no native ones for dotnet. There is an ODBC driver but requires a level of environment setup that is not always easy possible on cloud infrastructure. Which is why I was writing a a native dotnet connector, however stuck on determining what the contract is for SQL warehouses and how it differs from normal clusters