Error while running Job sparkSession is null while trying to executeCollectResult

Community Discussions

Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.

Hi,

I have started getting following error while running jobs in databrick. It started failing since last few days. Recently I have migrated to unity catalogue, no other change was made recently. I am running on DBR 13.3 LTS.

com.google.common.util.concurrent.ExecutionError: java.lang.AssertionError: assertion failed: sparkSession is null while trying to executeCollectResult

This is streaming code which runs jobs and its scheduled to run after every 2 hours.

(spark.readStream.format("delta")

.option("useNotification","true")

.option("includeExistingFiles","true")

.option("allowOverwrites",True)

.option("ignoreMissingFiles",True)

.option("maxFilesPerTrigger", 100)

.option("ignoreChanges","true")

.load(bronze_path)

.writeStream

.option("checkpointLocation", my_checkpoint_path)

.trigger(once=True)