Disable Databricks-generated error messages

alsetr — Fri, 09 May 2025 10:54:11 GMT

Since Databricks Runtime 12.2 Databricks started to wrap spark exceptions in their own exceptions.

https://learn.microsoft.com/en-us/azure/databricks/error-messages/

While for some users it might be handy, for our team it is not convinient, as we cannot see original exception, check whats going on in source code etc. When I put these stacktrace to IntelliJ, I cannot find such lines of code.

For example Databricsk say QueryExecutionErrors.scala:3372, but this file in Spark source code has only 2700 LoC and EXECUTOR_BROADCAST_JOIN_OOM cannot be found in Spark source code.

Could you please advise how to disable Databricsk error wrapping and get raw Spark error?

Re: Disable Databricks-generated error messages

-werners- — Fri, 09 May 2025 11:59:27 GMT

Databricks does not use vanilla spark.
They added optimizations like the AQE, unity catalog etc.
So looking for the error in the spark source code will not always work (in a lot of cases it will)

topic Disable Databricks-generated error messages in Data Engineering

Disable Databricks-generated error messages

Re: Disable Databricks-generated error messages