databricks-jdbc lists `spark_catalog` among catalogs for Standard tier Azure workspace

[HakanNordgren]

databricks-jdbc lists `spark_catalog` among catalogs for Standard tier Azure workspace. The UI lists `hive_metastore`. It would be better if these two were consistent.

[Kaniz_Fatma]

Hi @HakanNordgren, Let’s address the inconsistency between spark_catalog and hive_metastore in Databricks.

Here’s what we know:

1. spark_catalog:

   • This catalog is associated with Databricks and is used for managing metadata related to tables, views, and databases within Databricks.
   • It provides a unified view of data across various data sources and allows seamless querying using SQL.
   • The spark_catalog is an integral part of Databricks’ functionality.

2. hive_metastore:

   • The Hive Metastore is a component of Apache Hive, which is widely used for managing metadata in the Hadoop ecosystem.
   • It stores information about tables, partitions, and schemas.
   • Databricks leverages the Hive Metastore for compatibility and integration with existing Hive workloads.

Now, let’s discuss consistency and recommendations:

• Consistency:

  • It’s true that the UI and JDBC connection details should align to avoid confusion.
  • The UI displaying hive_metastore while JDBC lists spark_catalog can indeed be perplexing for users.

Thank you for flagging this.

[HakanNordgren]

@Kaniz_Fatma : Could I convince you to add the above as a bug to the backlog?

[HakanNordgren]

Also, @Kaniz_Fatma : Is it possible to have both a `hive_metastore` catalog and a `spark_catalog` catalog.

[MichTalebzadeh]

short answer is no

Databricks uses a single catalog -> Unity Catalog for managing tables created with Hive or Spark SQL, simplifying things. In spark sql you can do

spark-sql (default)> show catalogs;
spark_catalog

HTH