cancel
Showing results for 
Search instead for 
Did you mean: 
Community Platform Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results for 
Search instead for 
Did you mean: 

Error while running Job sparkSession is null while trying to executeCollectResult

sanjay
Valued Contributor II

Hi,

I have started getting following error while running jobs in databrick. It started failing since last few days. Recently I have migrated to unity catalogue, no other change was made recently. I am running on DBR 13.3 LTS.

 

com.google.common.util.concurrent.ExecutionError: java.lang.AssertionError: assertion failed: sparkSession is null while trying to executeCollectResult

 
This is streaming code which runs jobs and its scheduled to run after every 2 hours.
 
(spark.readStream.format("delta")
.option("useNotification","true")
.option("includeExistingFiles","true")
.option("allowOverwrites",True)
.option("ignoreMissingFiles",True)
.option("maxFilesPerTrigger", 100)
.option("ignoreChanges","true")
.load(bronze_path)
.writeStream
.option("checkpointLocation", my_checkpoint_path)
.trigger(once=True)
.foreachBatch(foreachBatchFunction)
.start()
.awaitTermination()
)

Appreciate any help.

Regards,

Sanjay

1 REPLY 1

Witold
Contributor III

Although it won't be the root cause of your issue, one thing worth mentioning is that `.trigger(once=True)` is obsolete by now and was replaced by `availableNow=True`

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group