Error while running Job sparkSession is null while trying to executeCollectResult

[sanjay]

Hi,

I have started getting following error while running jobs in databrick. It started failing since last few days. Recently I have migrated to unity catalogue, no other change was made recently. I am running on DBR 13.3 LTS.

com.google.common.util.concurrent.ExecutionError: java.lang.AssertionError: assertion failed: sparkSession is null while trying to executeCollectResult

This is streaming code which runs jobs and its scheduled to run after every 2 hours.

(spark.readStream.format("delta")

.option("useNotification","true")

.option("includeExistingFiles","true")

.option("allowOverwrites",True)

.option("ignoreMissingFiles",True)

.option("maxFilesPerTrigger", 100)

.option("ignoreChanges","true")

.load(bronze_path)

.writeStream

.option("checkpointLocation", my_checkpoint_path)

.trigger(once=True)

.foreachBatch(foreachBatchFunction)

.start()

.awaitTermination()

)

Appreciate any help.

Regards,

Sanjay

[Kaniz_Fatma]

Hi @sanjay, 

• Ensure that your Spark configuration is correctly set up. Sometimes invalid configurations can cause unexpected issues. Review your Spark settings and verify that everything is in order.

[Witold]

Although it won't be the root cause of your issue, one thing worth mentioning is that `.trigger(once=True)` is obsolete by now and was replaced by `availableNow=True`