How to Connect Pyspark to Unity Catalog on Kubernetes with Data Stored in MinIO?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-28-2024 09:04 PM
https://stackoverflow.com/questions/79177219/how-to-connect-spark-to-unity-catalog-on-kubernetes-wit...
I have posted the question on stack overflow.
I am trying to register catalog using pyspark.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-06-2024 08:52 AM
Thanks for your question!
Although it shouldn't be necessary, could you please try the following:
-
Set the
spark.databricks.sql.initial.catalog.nameconfiguration tomy_catalogin your Spark session to ensure the correct catalog is initialized. -
Use
current_catalog()to print the active catalog before executing the query. This will help verify the catalog in use. -
Print the query's
explain()plan in extended mode and check if any other catalog is being referenced. Even though you're using the three-level notation, this step can help confirm there are no unintended catalog overrides.
Let me know if these steps reveal any discrepancies or require further exploration!