topic Re: Issues with Querying and writing into a External Storage in a new databricks workspace in Data Governance

Issues with Querying and writing into a External Storage in a new databricks workspace

dipanjannet — Tue, 20 May 2025 11:27:27 GMT

Hello All,

We have recently created a databricks with UC enabled. All the VNET Pairing is completed. Now, while I am trying to create a table [Managed] under a catalog / schema - I am not able to query the table. I am getting below error

org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient

Also, while I am trying to write a Dataframe into an External location, I am seeing the same issue as mentioned. Any idea what is wrong here? Please note - as I will be using Unity catalog, nothing has been done (configuration) with respect to hive_metastore. Is this some hive_metastore related configuration issue ?

my write code

# Write the DataFrame to the specified Delta table

df_out.write.format("delta").mode("overwrite").option(

"path", "abfss://container@storageaccountname.dfs.core.windows.net/test_data/tbd/"

).saveAsTable("schema_name.table_name")

Re: Issues with Querying and writing into a External Storage in a new databricks workspace

Shua42 — Tue, 20 May 2025 17:08:20 GMT

Hi @dipanjannet ,

This could be due to the code falling back to the hive metastore, which could be due to a number of reasons, but the first thing that caught my eye was that the table name in your saveAsTable() call isn't fully qualified. Can you add the catalog name to the function call like saveAsTable("catalog_name.schema_name.table_name"), and if you haven't created a catalog yet I would start there.

Re: Issues with Querying and writing into a External Storage in a new databricks workspace

dipanjannet — Tue, 20 May 2025 17:16:30 GMT

Hello @Shua42 - Actually I was using catalog only. The code is little modified one. Can you please help me understand - what do you mean by this - "This could be due to the code falling back to the hive metastore, "

Re: Issues with Querying and writing into a External Storage in a new databricks workspace

Shua42 — Tue, 20 May 2025 18:00:08 GMT

The legacy Databricks Hive metastore is used as a fallback for queries when Unity Catalog is unavailable or when specific external locations are configured with fallback mode enabled, so it could be something in the configuration here. Were you able to create a catalog and schema in UC before writing the table?