topic Re: How to remove legacy hive metasore under catalog section ? in Data Governance

How to remove legacy hive metasore under catalog section ?

DSoni — Thu, 16 Jan 2025 15:34:20 GMT

Respected Concern,

We are currently using unity catalog in our workspace but we still see legacy hive_metastore under catalog section. Is there any way we can remove it. Because the issue we are facing is our cluster still somehow tries to connect with hive metastore client even though we are using unity catalog and it's throwing an exceptions continously.

It would be great if someone can help resolve this issue.

Thanks 🙂

Re: How to remove legacy hive metasore under catalog section ?

Alberto_Umana — Fri, 17 Jan 2025 12:58:31 GMT

Hello @DSoni,

Removing the hive_metastore catalog from your environment is not currently supported. Can you share more details on the issue you are having? What cluster mode access are you using? Also what are the errors?

Re: How to remove legacy hive metasore under catalog section ?

DSoni — Fri, 17 Jan 2025 15:09:57 GMT

Thank you for your response.

I understand that removing the hive_metastore catalog is not supported. To provide more context, here are the details of the issue I am encountering:

We are using Unity Catalog in my workspace and no longer need the hive_metastore.

Cluster Mode Access: Multi Node (Shared) (Single Job Cluster)
Mechanism: Structured Streaming
Cloud Provider: AWS
DB Version: 15.4 LTS

I hope this information helps in diagnosing the issue. Please let me know if you need any further details or logs.

Thank you for your assistance.

Re: How to remove legacy hive metasore under catalog section ?

DSoni — Fri, 17 Jan 2025 15:10:28 GMT

Error we encounter is below:

Errors:

Caused by: org.datanucleus.exceptions.NucleusException: Attempt to invoke the "HikariCP" plugin to create a ConnectionPool gave an error: Failed to initialize pool: Could not connect to address=(host=xyz.region.rds.amazonaws.com)(port=<port>)(type=master): Could not connect to xyz.region.rds.amazonaws.com:<port>: Connection reset

at org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:232)
at org.datanucleus.store.rdbms.ConnectionFactoryImpl.initialiseDataSources(ConnectionFactoryImpl.java:117)
at org.datanucleus.store.rdbms.ConnectionFactoryImpl.<init>(ConnectionFactoryImpl.java:82)
... 127 more

Caused by: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: Could not connect to address=(host=xyz.region.rds.amazonaws.com)(port=<port>)(type=master): Could not connect to xyz.region.rds.amazonaws.com:<port>: Connection reset

at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:512)
at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:105)
at com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:71)
at org.datanucleus.store.rdbms.connectionpool.HikariCPConnectionPoolFactory.createConnectionPool(HikariCPConnectionPoolFactory.java:176)
at org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:213)
... 129 more

Caused by: java.sql.SQLNonTransientConnectionException: Could not connect to address=(host=xyz.region.rds.amazonaws.com)(port=<port>)(type=master): Could not connect to xyz.region.rds.amazonaws.com:<port>: Connection reset

at org.mariadb.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:73)
at org.mariadb.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:197)
at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1404)
at org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:635)
at org.mariadb.jdbc.MariaDbConnection.newConnection(MariaDbConnection.java:150)
at org.mariadb.jdbc.Driver.connect(Driver.java:89)
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:341)
at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:506)
... 133 more

Caused by: java.sql.SQLNonTransientConnectionException: Could not connect to xyz.region.rds.amazonaws.com:<port>: Connection reset

at org.mariadb.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:73)
at org.mariadb.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:188)
at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:588)
at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1399)
... 140 more

Caused by: java.net.SocketException: Connection reset

at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at org.mariadb.jdbc.internal.io.input.ReadAheadBufferedStream.fillBuffer(ReadAheadBufferedStream.java:131)
at org.mariadb.jdbc.internal.io.input.ReadAheadBufferedStream.read(ReadAheadBufferedStream.java:104)
at org.mariadb.jdbc.internal.io.input.StandardPacketInputStream.getPacketArray(StandardPacketInputStream.java:247)
at org.mariadb.jdbc.internal.io.input.StandardPacketInputStream.getPacket(StandardPacketInputStream.java:218)
at org.mariadb.jdbc.internal.com.read.ReadInitialHandShakePacket.<init>(ReadInitialHandShakePacket.java:89)
at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:540)
... 141 more

Re: How to remove legacy hive metasore under catalog section ?

DSoni — Wed, 22 Jan 2025 14:27:11 GMT

Hello @Alberto_Umana ,
Would you mind providing an update on the information I provided ?
It would be really appreciated !
Thanks 🙂

Re: How to remove legacy hive metasore under catalog section ?

Alberto_Umana — Wed, 22 Jan 2025 15:01:06 GMT

Hi @DSoni,

Thanks for the details you have shared!

Couple of question, are you interacting with any table in hive_metastore path?

Do you have a private link setup?

Based on the error there might be a limit in the pool of the RDS of hive metastore in your AWS instance, can you check RDS metrics to validate and increase the limit?

Re: How to remove legacy hive metasore under catalog section ?

DSoni — Wed, 22 Jan 2025 20:36:26 GMT

Hello @Alberto_Umana ,
I consulted with our DevOps and we are not interacting with any table in hive_metasore , neither we have any private link setup. Also we are not using RDS !

Re: How to remove legacy hive metasore under catalog section ?

Alberto_Umana — Thu, 23 Jan 2025 13:52:41 GMT

Hi @DSoni,

By default hive_metastore uses an RDS in AWS, in the failure you can see: xyz.region.rds.amazonaws.com which I think you redacted, that is the host with the issue. I would require more details to check hive_metastore comes into picture if you mentioned no hive_metastore table is being used.

Was the error observed during a workflow run? or during a manual query?

Re: How to remove legacy hive metasore under catalog section ?

DSoni — Thu, 23 Jan 2025 14:00:47 GMT

Hello @Alberto_Umana ,
We are running workflow and in driver logs we see this exception. We are using structured streaming in our work. Everything is in UC.

Re: How to remove legacy hive metasore under catalog section ?

bhanu_dp — Tue, 11 Mar 2025 06:06:22 GMT

Were you able to get any solution/workaround to your requirement?