Databricks Community

TjommeV-Vlaio

Hi,We're running DBR 14.3 on a shared multi-node cluster.When checking the metrics of the driver, I see that the Memory utilization and Memory swap utilization are increasing a lot and are almost never decreasing. Even if no processes are running any...

TjommeV-Vlaio

I'll check this out. My goal is to see which notebooks/processes are consuming large amounts of driver memory (without releasing it) as this might indicate there is a memory leak or coding contains some non-parallel code that needs to be resolved.

TjommeV-Vlaio

Found the web terminal doc: Run shell commands in Azure Databricks web terminal - Azure Databricks | Microsoft LearnUnfortunately, we're running shared clusters on DBR 14.3, so no web terminal supportRunning %sh htop from a notebook does not align wi...

TjommeV-Vlaio

Do I do this from Databricks or from Azure? To do it from Azure, I'm missing the credentials to connect.From Databricks I don't know how to do this

TjommeV-Vlaio

Hi Walter,This overview shows the consumption by node, not by process.Thread dump and heap histogram do not seem to provide usefull information (for my issue)

TjommeV-Vlaio · 11-07-2024

Hi,We switched to using the pymssql library.Incomplete code sample to give you an idea:import pymssql # strParamList is comma separated list of @Param="some value" sqlExecuteStoredProc = f"exec {strStoredProcName} {strParamList}" _MetaDBConnection =...

Databricks Community

User Stats

User Activity

Which process is eating up my driver memory?

Re: Which process is eating up my driver memory?

Re: Which process is eating up my driver memory?

Re: Which process is eating up my driver memory?

Re: Which process is eating up my driver memory?

Re: Running Stored Procedures on a Multi-node/ Shared Cluster