GRCL
New Contributor III

Here is the resolution for my particular case, it should help you for the first topic issue.

We were using those 2 lines of code in a library (whl file library in Databricks cluster global init script) created by our own (needed to do that in v9.1 because spark session was not available without this code) :

sc = SparkContext.getOrCreate()
spark = SparkSession(sc)

On v11.3 we found that we have now to replace the 2 lines of code by only one (get the active session), previous method seems deprecated :

spark = SparkSession.getActiveSession()

And everything works well !

View solution in original post