topic Re: Executor OOM Error with AQE enabled in Data Engineering

Executor OOM Error with AQE enabled

alsetr — Fri, 09 May 2025 11:22:07 GMT

We have Databricks Spark Job. After migration from Databricks Runtime 10.4 to 15.4 one of our Spark jobs which uses broadcast hint started to fail with error:

```
ERROR Executor: Exception in task 2.0 in stage 371.0 (TID 16912)
org.apache.spark.memory.SparkOutOfMemoryError: [EXECUTOR_BROADCAST_JOIN_OOM] There is not enough memory to build the broadcast relation LongToUnsafeRowMap. Relation Size = 1462.4 MiB. Total memory used by this task = 1526.4 MiB. Executor Memory Manager Metrics: onHeapExecutionMemoryUsed = 2.4 GiB, offHeapExecutionMemoryUsed = 0.0 B, onHeapStorageMemoryUsed = 472.5 MiB, offHeapStorageMemoryUsed = 0.0 B. [sparkPlanId: Some(44226)] SQLSTATE: 53200
```
This job fails regardless resources we use, it fails even with Standard_D8s_v3 worker nodes, which has 32GB RAM.
Also before the error we have log message which show that there is enough memory.
```
INFO MemoryStore: Block broadcast_188 stored as values in memory (estimated size 359.3 KiB, free 24.0 GiB)
```

Looks like this is Adaptive Query Execution issue, as disabling it solves the problem.

Could anybody advise how to overcome this issue without disabling AQE?

Re: Executor OOM Error with AQE enabled

-werners- — Fri, 09 May 2025 11:54:53 GMT

have you tried removing the broadcast hint?
In recent versions of databricks runtime a lot of optimizations have been added.
Also: do you use the photon engine?

Re: Executor OOM Error with AQE enabled

alsetr — Fri, 09 May 2025 12:06:01 GMT

no, we dont want to remove broadcast hint, as it works without problems in DBR 10.4, and there is a lot of memory availble for it.

Re: Executor OOM Error with AQE enabled

-werners- — Fri, 09 May 2025 12:12:18 GMT

ok, that is up to you.

An executor will not be able to take all the ram.
you can try to work with the spark.executor parameters.

Re: Executor OOM Error with AQE enabled

alsetr — Tue, 01 Jul 2025 11:06:16 GMT

Looks like the reason of error is a bug in new Databricks feature which is called executor-side broadcast (ebj, executor broadcast join) which was introduced in DBR 11.3.

Unfortunately could not find the way how to disable this feature, so keeping AQE disable so far.