iceberg

[Mohsen]

Hi fellas
i am working on databricks using iceberg
at first i have configured my notebook as below
spark.conf.set("spark.sql.catalog.spark_catalog","org.apache.iceberg.spark.SparkCatalog")
spark.conf.set("spark.sql.catalog.spark_catalog.type", "hadoop")
spark.conf.set("spark.sql.catalog.spark_catalog.warehouse", "dbfs:/TPCDSIceberg/Data/")
then i created an iceberg tables

spark.sql(s"CREATE DATABASE IF NOT EXISTS bronzedb") spark.sql(s"DROP TABLE IF EXISTS bronzedb.bronzeTable") // Create an empty Delta table in the bronze layer spark.sql( s""" |CREATE TABLE bronzedb.bronzeTable($schemas) |USING $sinkFormat |location 'dbfs:/TPCDSIceberg/Data/bronzedb/bronzeTableName' |""".stripMargin )

and then i tried to load this table

val icebergDf = spark.read.format("iceberg").load(s"bronzedb.bronze_call_center")

but i get this Exception

RuntimeMetaException: Failed to connect to Hive Metastore Caused by: RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient Caused by: InvocationTargetException: Caused by: MetaException: Version information not found in metastore.

[Kaniz_Fatma]

Hi @Mohsen, 

• The exception "RuntimeMetaException: Failed to connect to Hive Metastore" occurs because the Hive metastore cannot find the version information.

• To resolve the issue, follow the steps below:

  - Set up a cluster with spark.sql.hive.metastore.version set to 2.1.1 and spark.sql.hive.metastore.jars set to maven.
  - Set the following properties:
      - spark.hadoop.javax.jdo.option.ConnectionURL
      - spark.hadoop.javax.jdo.option.ConnectionUserName
      - spark.hadoop.javax.jdo.option.ConnectionPassword
      - spark.hadoop.javax.jdo.option.ConnectionDriverName (set it to com.microsoft.sqlserver.jdbc.SQLServerDriver)
  - Find the location of the downloaded metastore JARs using the command grep "Downloaded metastore jars to" /databricks/driver/logs/log4j-active.log.
  - Copy the downloaded JARs to DBFS using the DBFS mount point in the local file system of the driver node.
  - Replace datanucleus-rdbms 4.1.7 with datanucleus-rdbms 4.1.16 by deleting the old JAR and downloading the new JAR.
• < UNK> Following these steps, you should be able to resolve the issue and connect to the Hive metastore successfully.