topic Connect Spark Cluster to SQL Endpoint in Administration & Architecture

Connect Spark Cluster to SQL Endpoint

carroll_q2 — Wed, 28 Jun 2023 11:00:57 GMT

Hello! Is it possible to retrieve data from a SQL Endpoint in the Databricks SQL persona using the Data Science and Engineering persona? In other words, I would like to use pyspark in DS&E to query a table in Databricks SQL.

#DatabricksSQL

#DatabricksDS&E

#ConnectClustersToEndPoints

Re: Connect Spark Cluster to SQL Endpoint

-werners- — Wed, 28 Jun 2023 12:12:23 GMT

you do not need the sql warehouse itself for that. for ds & e you need a classic cluster (not a sql endpoint) anyway so you can easily read the tables from the metastore using spark.read.table().

Connecting the sql endpoint to the ds cluster seems odd, because what part of the query plan will be executed by the sql endpoint and what part by the ds cluster?

Right now you can already use a sql endpoint for sql notebooks.

Re: Connect Spark Cluster to SQL Endpoint

carroll_q2 — Wed, 28 Jun 2023 12:38:29 GMT

@-werners- Thank you for the response! You make some valid points. Specifically, your recommendation to use spark.read.table() was the solution.