cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

How to import SqlDWRelation from com.databricks.spark.sqldw

WillJMSFT
New Contributor III

Hello, All - I'm working on a project using the SQL DataWarehouse connector built into Databricks (https://docs.databricks.com/data/data-sources/azure/synapse-analytics.html). From there, I'm trying to extract information from the logical plan / logical relation which will help me identify the table and database (or at least JDBC URL) that is being used to query the data.

I believe I need to cast from BaseRelation to SqlDWRelation after extracting the relation from the logical plan as seen below.

val df = spark.read.format("com.databricks.spark.sqldw")
.option("url", sqlDwUrl)
.option("tempDir", tempDir)
.option("forwardSparkAzureStorageCredentials", "true")
.option("dbTable", tableName)
.load()
val logicalPlan = df.queryExecution.logical
val logicalRelation = logicalPlan.asInstanceOf[LogicalRelation]
val sqlBaseRelation = logicalRelation.relation

However, after doing so, I try to cast `sqlBaseRelation.asInstanceOf[SqlDWRelation]` and it results in an error that looks like:

object SqlDWRelation in package sqldw cannot be accessed in package com.databricks.spark.sqldw

Edit: The above error makes sense since it's a private class 😞

When looking at the jar provided in the workspace for the SQL DW connector, I can see SqlDWRelation as a (edit public ) private class but it's not available via tab / autocomplete inside a Databricks Notebook.

Has anyone here ever attempted to work with a SqlDWRelation and extract information about the table and database names for the given logicalrelation? Thank you for any guidance!

In my environment, I'm using:

  • Azure Databricks
  • Databricks Runtime 9.1
  • Scala Notebook
  • Azure Synapse SQL Pools
1 ACCEPTED SOLUTION

Accepted Solutions

-werners-
Esteemed Contributor III

Lol so I was typing a response but now I see you edited your post.

A private method cannot be called indeed.

I worked with the connector but only to run queries and such, not to fetch metadata.

But the golden rule is: if autocomplete shows it, you can use it.

Maybe with a lot of effort because of missing docs, but it can be done.

View solution in original post

6 REPLIES 6

Kaniz
Community Manager
Community Manager

Hi @ WillJMSFT! My name is Kaniz, and I'm the technical moderator here. Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question first. Or else I will get back to you soon. Thanks.

-werners-
Esteemed Contributor III

Have you tried to pass as simple query to Synapse to see if that works?

WillJMSFT
New Contributor III

@Werner Stinckens​  Thanks for the reply! The SQL DW Connector itself is working just fine and I can retrieve the results from the SQL DW. I'm trying to extract the metadata (i.e. the Server, Database, and Table name) from the logical plan (or through any other means).

-werners-
Esteemed Contributor III

Lol so I was typing a response but now I see you edited your post.

A private method cannot be called indeed.

I worked with the connector but only to run queries and such, not to fetch metadata.

But the golden rule is: if autocomplete shows it, you can use it.

Maybe with a lot of effort because of missing docs, but it can be done.

WillJMSFT
New Contributor III

Ah! Yes! Sorry about that! I realized this morning that it was a private class and that's why I wasn't seeing it show up in autocomplete.

The next step is to figure out how we can still access that data even when it's using a BaseRelation and not just the (private) SqlDWRelation.

I really appreciate your replies, @werners!

Hi @Will Johnson​ ,

If @Werner Stinckens​ 's reply helped you to solved this issue, could you select it as best response? it will help to move it to the top.

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.