The databricks jdbc driver has a memory leak

gf — Thu, 26 Dec 2024 06:05:37 GMT

https://community.databricks.com/t5/community-platform-discussions/memory-leak/td-p/80756 My question is the same as above Unable to upload pictures, I had to dictate

Question from ResultFileDownloadMonitor. M_requestList parameters
Because is ResultFileDownloadMonitor getResultFileDownloadMonitor method, so this class into a singleton, and cannot be GC
The m_requestList variable is a HashMap, and the addDownloadTask method is invoked to add elements to the m_requestList whenever the sql query is executed
The problem is that the m_requestList object only has the logic to add and modify elements, but no logic to delete elements, which is weird
After testing, we found that there are more than 120,000 elements in the m_requestList from the dump file, which is only 3 days of data volume
Although the size of each element is small, the amount is too large, and over time there will always be problems
I checked the driver for the latest 2.6.40-patch-1 release and found that the problem still exists

Therefore, I would like to confirm the following questions:
1. Will this issue be fixed in a future release?
2. Can the stopped object in m_requestList be deleted? Can I periodically clean KV whose value is true in m_requestList using reflection?

Re: The databricks jdbc driver has a memory leak

Walter_C — Thu, 26 Dec 2024 14:37:07 GMT

Hello @gf thanks for your question, it seems that this has been reported with Simba, but no fix has been provided yet, as a temporary workaround, you can consider using reflection to periodically clean up the m_requestList by removing KV pairs whose value is true. However, this should be done with caution and thorough testing to ensure it does not introduce new issues.

topic The databricks jdbc driver has a memory leak in Get Started Discussions

The databricks jdbc driver has a memory leak

Re: The databricks jdbc driver has a memory leak