Databricks Community

wojciech_jakubo · 06-21-2023

Hi databricks/spark experts!I have a piece on pandas-based 3rd party code that I need to execute as a part of a bigger spark pipeline. By nature, pandas-based code is executed on driver node. I ran into out of memory problems and started exploring th...

wojciech_jakubo · 08-16-2023

Hi, How did you infer from this image that driver memory would be 21.4 GB? Shouldn't it be 10.4 GB? Also, if a memory is allocated to JVM related activities, can this memory can be also utilized from python? Meaning if I have 21.4 gb for JVM, does it...

wojciech_jakubo · 08-16-2023

Hi Tharun-Kumar.Thanks for your answer. I get that all these activities you listed are required for cluster to function correctly. But 40 gigs of ram for that? That looks way too much imo... especially that all these activates are also done on much s...

wojciech_jakubo · 07-27-2023

I think we really appreciate you caring about our challenges with complexity and overhead. But the thing is that each org is different and has different needs. And this constraint is a real deal breaker. Because there is only one metastore per region...

wojciech_jakubo · 07-27-2023

I very much like the features that come with unity catalog. But at the same time I find it extremally challenging to implement this in a big organization in its current form due to 1-1 relation to AAD tenant and 1 metastore constraint. We have one AA...

Databricks Community

User Stats

User Activity

Question about monitoring driver memory utilization

Re: Question about monitoring driver memory utilization

Re: Question about monitoring driver memory utilization

Re: Unity Catalog - multiple metastore in same region

Re: Unity Catalog - multiple metastore in same region