Hi Lou,

thanks for your detailed answer!

So far I'm trying option 2, but exchanging so many dependencies seems like a strange thing to do and feels like a g-ame of whack-a-mole as new build errors pop up. If the dependency situations gets resolved with DBR 17, it might be acceptable.

I guess I'll try option 3 as well and cross-build my project.

What do you mean with option 1? Not using Databricks Connect at all? In that case, do I just use normal Spark (3.5.2) dependencies to run my job on the cluster? Do I use Spark Connect's SparkSession or plain old SparkContext?

 

View solution in original post