Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-08-2023 06:39 AM
Here is the resolution for my particular case, it should help you for the first topic issue.
We were using those 2 lines of code in a library (whl file library in Databricks cluster global init script) created by our own (needed to do that in v9.1 because spark session was not available without this code) :
sc = SparkContext.getOrCreate()
spark = SparkSession(sc)On v11.3 we found that we have now to replace the 2 lines of code by only one (get the active session), previous method seems deprecated :
spark = SparkSession.getActiveSession()And everything works well !