Hi @Retired_mod,

I ensured that I was using the correct Spark version that matched the version of my databricks runtime, which was the same. But I tried use the Spark version 3.5.1 locally in the .sbt dependencies, then this worked, kind strange.

Anyway thank you very much @Retired_mod, your comment was very helpfully.