Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-04-2025 02:45 AM
When you say:
"I have a Databricks pyspark query that has been running fine for the last two weeks but am now getting the following error despite no changes to the query: OutOfMemoryError: GC overhead limit exceeded."
Can you tell us how do you execute your query?
Is it a notebook that is always executed on the same 'all purpose compute"?
If yes, I think you are facing the same issue than me. When I run my notebook once and then let time to the compute to shut down, it is ok. But, if I run my notebook several times without letting time for the cluster to shutdown, then, after several occurences it crash. (I don't use cache at all, so no memory leak here)