Error while running Job sparkSession is null while trying to executeCollectResult

sanjay — Fri, 28 Jun 2024 11:52:00 GMT

Hi,

I have started getting following error while running jobs in databrick. It started failing since last few days. Recently I have migrated to unity catalogue, no other change was made recently. I am running on DBR 13.3 LTS.

com.google.common.util.concurrent.ExecutionError: java.lang.AssertionError: assertion failed: sparkSession is null while trying to executeCollectResult

This is streaming code which runs jobs and its scheduled to run after every 2 hours.

(spark.readStream.format("delta")

.option("useNotification","true")

.option("includeExistingFiles","true")

.option("allowOverwrites",True)

.option("ignoreMissingFiles",True)

.option("maxFilesPerTrigger", 100)

.option("ignoreChanges","true")

.load(bronze_path)

.writeStream

.option("checkpointLocation", my_checkpoint_path)

.trigger(once=True)

.foreachBatch(foreachBatchFunction)

.start()

.awaitTermination()

)

Appreciate any help.

Regards,

Sanjay

Re: Error while running Job sparkSession is null while trying to executeCollectResult

Witold — Mon, 01 Jul 2024 12:39:32 GMT

Although it won't be the root cause of your issue, one thing worth mentioning is that `.trigger(once=True)` is obsolete by now and was replaced by `availableNow=True`

topic Re: Error while running Job sparkSession is null while trying to executeCollectResult in Get Started Discussions

Error while running Job sparkSession is null while trying to executeCollectResult

Re: Error while running Job sparkSession is null while trying to executeCollectResult