Connect Spark Cluster to SQL Endpoint

[carroll_q2]

Hello! Is it possible to retrieve data from a SQL Endpoint in the Databricks SQL persona using the Data Science and Engineering persona?  In other words, I would like to use pyspark in DS&E to query a table in Databricks SQL.

#DatabricksSQL

#DatabricksDS&E

#ConnectClustersToEndPoints

[-werners-]

you do not need the sql warehouse itself for that.  for ds & e you need a classic cluster (not a sql endpoint) anyway so you can easily read the tables from the metastore using spark.read.table().

Connecting the sql endpoint to the ds cluster seems odd, because what part of the query plan will be executed by the sql endpoint and what part by the ds cluster?

Right now you can already use a sql endpoint for sql notebooks.

[carroll_q2 · Accepted solution]

@-werners- Thank you for the response!  You make some valid points.  Specifically, your recommendation to use spark.read.table() was the solution.