@ls Agree, it doesn't seem to be fixed. Maybe on DBR 16 memory management is better optimized, hence I'd like to suggest going through the methods mentioned earlier in this post:

• Memory Profiling: Try freezing the dataset that reproduces the problem and profile memory usage on the Python side. Look for issues like data distribution problems or skewness. Tools like memory-profiler might help.

• Error Isolation: If possible, isolate the query and dataset causing the issue. Translating the Python code to Scala can help revealing the underlying exception.(This will not be straightforward). Instead, works needs to be done to capture the Python side thread dumps upon OOM error (traceback and/or faulthandler modules can become useful)

• Logs and Metrics: Check if there are any OOM-related messages in the executor logs. Additionally, analyze task metrics per executor in the Spark UI for any anomalies.

• Cluster Resources: As a last resort, you could temporarily increase cluster resources to allow the job to complete. This approach can help you gather insights from the Spark UI metrics to understand why the current cluster size is insufficient for processing the dataset.

Additional Resources:

https://pypi.org/project/memory-profiler/ 

https://www.databricks.com/blog/2022/11/30/memory-profiling-pyspark.html

If there are any challenges with any of the above steps,  e.g.: with the memory profiling, you may raise a Support Ticket, so that we can better assist you in identifying the root cause and ultimately fixing the OOMs.