The issue with SQL-endpoint + external hive metastore
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-17-2022 11:44 PM
Hi
We have faced an issue recently (everything worked well 1 week before)
We are using exnernal hive metastore:
"spark.hadoop.datanucleus.connectionPool.maxPoolSize" : "10"
"spark.sql.hive.metastore.jars" : "/dbfs/ivyjars/*"
"spark.sql.hive.metastore.version" : "3.1.0"and now it looks like all access for the metastore from sql-endpoints are stuck (I am observing a lot of queued session from query history like "show databases", "show table" and so on )
the version of sql-warehouse:
Channel Current (v 2022.40)We are using rds for hive metastore and I'm observing a lot of idle sessions such as:
idle,,,SET application_name = 'PostgreSQL JDBC Driver'
idle,,,SET application_name = 'PostgreSQL JDBC Driver'
idle,,,SET application_name = 'PostgreSQL JDBC Driver'
idle,,,SET application_name = 'PostgreSQL JDBC Driver'
idle,,,SET application_name = 'PostgreSQL JDBC Driver'
idle,,,SET application_name = 'PostgreSQL JDBC Driver'
idle,,,SET application_name = 'PostgreSQL JDBC Driver'
idle,,,SET application_name = 'PostgreSQL JDBC Driver'
idle,,,COMMITWhat is the issue?
P.S.
The restarting of the SQL -warehouse helps sometimes, but the problem still exists
- Labels:
-
Query History
-
SQL
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-18-2022 08:26 PM
Generally, this should not be a case. Please raise a support request to databricks