Hi @Malthe ,

This might be because of New DBR (18.0) GA release yesterday(January 2026 - Azure Databricks | Microsoft Learn). you might need to use a custom spark version by the time engineering team fixes this issue in DBR. Below is the response from Databricks Support for similar sort of problem.

"

There was a DBR release on 7th August (14.3.10 -> 14.3.11).
Our engineering team identified the issue and the fix is scheduled to be deployed on September 16th.

Until the fix is deployed, You can use the below custom spark image version in your cluster.

enter the below in the Custom Spark Version: The custom image provided was the old DBR prior to 8th August.
custom:release__14.3.x-snapshot-scala2.12__databricks-universe__14.3.10__9b6cd4f__debafb7__jenkins__1cbb705__format-3

"

link to instruction how to enable definition of custom spark version 

Run a custom Databricks Runtime on your cluster - Databricks