cancel
Showing results for 
Search instead for 
Did you mean: 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results for 
Search instead for 
Did you mean: 

Access Foreign Catalog using Python in Notebook

kashy
New Contributor III

Hello - I have a foreign catalog which I can access fine in SQL.  However, I can't access it from from python notebook.

i.e. this works just fine if I have notebook using a Pro SQL Warehouse

%sql
USE CATALOG <my_foreign_catalog_name>;
USE SCHEMA public;
SELECT * from bookings;

 

However, this, running on Shared Cluster 13.3 LTS (includes Apache Spark 3.4.1, Scala 2.12)
spark.sql("USE CATALOG <my_foreign_catalog_name>")
spark.sql("USE SCHEMA public")
display(spark.table("bookings"))
 
Gives me this error:
org.apache.spark.sql.connector.catalog.CatalogNotFoundException: Catalog 'my_foreign_catalog_name' plugin class not found: spark.sql.catalog.my_foreign_catalog_name is not defined
 
 
If I run:
display(spark.sql("SHOW CATALOGS"))
 
I can't see the foreign_catalogs I have created in Unity Catalog and it just shows me 'spark_catalog'
 
Any idea what needs tweaking?
1 ACCEPTED SOLUTION

Accepted Solutions

kashy
New Contributor III

thanks @Debayan 

I resolved this - I has to remove this option on the cluster and it works now

Enable credential passthrough for user-level data access

View solution in original post

2 REPLIES 2

Debayan
Databricks Employee
Databricks Employee

Hi, Are you using this in single user cluster? 

Also, please tag @Debayan with your next response so that I will get notified. 

kashy
New Contributor III

thanks @Debayan 

I resolved this - I has to remove this option on the cluster and it works now

Enable credential passthrough for user-level data access

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group