topic Re: Query Serverless SQL Warehouse from Spark Submit Job in Data Engineering

Query Serverless SQL Warehouse from Spark Submit Job

akhileshp — Thu, 07 Mar 2024 07:18:25 GMT

I am trying to load data from a table in SQL warehouse using spark.sql("SELECT * FROM <table>") in a spark submit job, but the job is failing with [TABLE_OR_VIEW_NOT_FOUND] The table or view . The same statement is working in notebook but not in a job. How to query the SQL warehouse from the spark job.

Re: Query Serverless SQL Warehouse from Spark Submit Job

akhileshp — Thu, 07 Mar 2024 11:23:30 GMT

I am running the spark job in databricks, so it is running in databricks runtime itself then why is the behaviour different with the notebook and a spark submit job

Re: Query Serverless SQL Warehouse from Spark Submit Job

Wojciech_BUK — Thu, 07 Mar 2024 12:12:41 GMT

Hey,
Is your table in Unity Catalog or in Hive Metastore ?
just my guess(es):

- use full table path Catalog_name.Schema_Name.Table_name (maybe you had some context saved when querying data interactively.

- if your table is in UC - is your Job Cluster running in UC enabled mode (single / shared) ?

From your post it is bit uncleare where and what you are running.

Re: Query Serverless SQL Warehouse from Spark Submit Job

akhileshp — Thu, 07 Mar 2024 12:18:23 GMT

The table is in hive_metastore and I am using the full path.

The job compute is created using an instance pool

Re: Query Serverless SQL Warehouse from Spark Submit Job

Wojciech_BUK — Thu, 07 Mar 2024 14:19:29 GMT

- when you query table manually and running job - do both those actions happens in same Databricks Workspace
- what is job configuration - who is job Owner or Run As Account -> do this principal/persona has access to the table ?

Re: Query Serverless SQL Warehouse from Spark Submit Job

akhileshp — Thu, 07 Mar 2024 15:45:03 GMT

Yes, Both are in the same workspace and querying same table

Yes the principle has all the access.

Also its working with spark python job, the problem is with spark submit job only.

My usecase is supported only in submit job since I am having multiple scripts in a zip file