Error while running Job sparkSession is null while trying to executeCollectResult
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-28-2024 04:52 AM
Hi,
I have started getting following error while running jobs in databrick. It started failing since last few days. Recently I have migrated to unity catalogue, no other change was made recently. I am running on DBR 13.3 LTS.
com.google.common.util.concurrent.ExecutionError: java.lang.AssertionError: assertion failed: sparkSession is null while trying to executeCollectResult
This is streaming code which runs jobs and its scheduled to run after every 2 hours.
(spark.readStream.format("delta")
.option("useNotification","true")
.option("includeExistingFiles","true")
.option("allowOverwrites",True)
.option("ignoreMissingFiles",True)
.option("maxFilesPerTrigger", 100)
.option("ignoreChanges","true")
.load(bronze_path)
.writeStream
.option("checkpointLocation", my_checkpoint_path)
.trigger(once=True)
.foreachBatch(foreachBatchFunction)
.start()
.awaitTermination()
)
Appreciate any help.
Regards,
Sanjay
