Issues with Querying and writing into a External Storage in a new databricks workspace
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-20-2025 04:27 AM
Hello All,
We have recently created a databricks with UC enabled. All the VNET Pairing is completed. Now, while I am trying to create a table [Managed] under a catalog / schema - I am not able to query the table. I am getting below error
org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
Also, while I am trying to write a Dataframe into an External location, I am seeing the same issue as mentioned. Any idea what is wrong here? Please note - as I will be using Unity catalog, nothing has been done (configuration) with respect to hive_metastore. Is this some hive_metastore related configuration issue ?
my write code
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-20-2025 10:08 AM
Hi @dipanjannet ,
This could be due to the code falling back to the hive metastore, which could be due to a number of reasons, but the first thing that caught my eye was that the table name in your saveAsTable() call isn't fully qualified. Can you add the catalog name to the function call like saveAsTable("catalog_name.schema_name.table_name"), and if you haven't created a catalog yet I would start there.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-20-2025 10:16 AM
Hello @Shua42 - Actually I was using catalog only. The code is little modified one. Can you please help me understand - what do you mean by this - "This could be due to the code falling back to the hive metastore, "
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-20-2025 11:00 AM
The legacy Databricks Hive metastore is used as a fallback for queries when Unity Catalog is unavailable or when specific external locations are configured with fallback mode enabled, so it could be something in the configuration here. Were you able to create a catalog and schema in UC before writing the table?
