topic How to ensure all resources are released when closing a Connection via DataBricks JDBC Driver? in Warehousing & Analytics

How to ensure all resources are released when closing a Connection via DataBricks JDBC Driver?

jon1 — Fri, 09 Dec 2022 10:34:47 GMT

Hi there,

I'm building a Sink that uses the Databricks JDBC Driver version 2.6.27. It appears that regardless of using a try-with-resources block and explicitly closing the Connection, the driver's internals are not releasing IdleConnectionEvictor threads which are spawned per Connection.

Connection.close() doesn't release the IdleConnectionEvictor thread:

app//com.databricks.client.jdbc42.internal.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)

So if you're spawning multiple Connections, you're left with these threads until Java process stop and restart.

Is this a bug or am I overlooking a step required by the DataBricks JDBC Driver?

P.S: I'd post code but no markdown syntax seems to work in this DataBricks Community forum for me. There's no code formatting options when starting a new discussion either.

Re: How to ensure all resources are released when closing a Connection via DataBricks JDBC Driver?

jon1 — Mon, 19 Dec 2022 14:12:20 GMT

Following appears to be have been resolved in JDBC driver version 2.6.29 and newer. See release notes and the bug fix [SPARKJ-614] After closing HTTP connections, the driver does not clean up native threads.

Re: How to ensure all resources are released when closing a Connection via DataBricks JDBC Driver?

RakeshSen01 — Tue, 13 Aug 2024 12:27:22 GMT

Still not fixed got 14k instances of com.databricks.client.jdbc42.internal.apache.http.impl.client.IdleConnectionEvictor 14,037 (0%) 673,776 B (0%) n/a And due to that application got to crashed. can you please help out as this is blocking to release databricks feature.