I've noticed a change in how Databricks handles unresolved column references in PySpark when using All-purpose compute (not serverless).
In Databricks Runtime 14.3 LTS, referencing a non-existent column like this:
df = spark.table('default.example').select('nonexistent_column')
...used to immediately throw an error like [UNRESOLVED_COLUMN.WITH_SUGGESTION], even before any actions like .show() or display() were called. This was helpful for catching typos or schema mismatches early.
However, sometime around April 2025, this eager error-throwing behavior disappeared in 14.3 LTS. Now, the error is only raised when a non-lazy action is triggered.
Note that in Runtime 16.4 LTS, the behavior has changed again. While it still doesn't raise an exception right away, I now see a large JSON-formatted error message logged to the console. Here is a screenshot
This seems like a partial restoration of the old behavior, where errors were visible right away, even if not thrown as exceptions.
I couldnโt find any mention of this change in the 16.4 LTS release notes. (By the way, Iโm ignoring 15.4 LTS because I havenโt used it much, but based on a quick test it seems to have the same problems as version 14.3 LTS.)
Is this new logging behavior intentional? Is there a config or setting to restore the old eager error-throwing behavior from earlier 14.3 LTS builds?
Final note: The only similar question I have found online is this thread: Lazy evaluation in serverless vs all purpose compu... - Databricks Community - 115459. It's also about the absence of an error message for lazy evaluation. However, the thread is specifically about serverless compute, whereas I am using all-purpose compute. So I think my issue must be different.
Thanks in advance!
