cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 

Connect Spark Cluster to SQL Endpoint

carroll_q2
New Contributor III

Hello! Is it possible to retrieve data from a SQL Endpoint in the Databricks SQL persona using the Data Science and Engineering persona?  In other words, I would like to use pyspark in DS&E to query a table in Databricks SQL.

#DatabricksSQL

#DatabricksDS&E

#ConnectClustersToEndPoints

1 ACCEPTED SOLUTION

Accepted Solutions

carroll_q2
New Contributor III

@-werners- Thank you for the response!  You make some valid points.  Specifically, your recommendation to use spark.read.table() was the solution. 

View solution in original post

2 REPLIES 2

-werners-
Esteemed Contributor III

you do not need the sql warehouse itself for that.  for ds & e you need a classic cluster (not a sql endpoint) anyway so you can easily read the tables from the metastore using spark.read.table().

Connecting the sql endpoint to the ds cluster seems odd, because what part of the query plan will be executed by the sql endpoint and what part by the ds cluster?

Right now you can already use a sql endpoint for sql notebooks.

carroll_q2
New Contributor III

@-werners- Thank you for the response!  You make some valid points.  Specifically, your recommendation to use spark.read.table() was the solution. 

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!