Eorror in perform a merge inside a streaming foreachbatch using the command: microBatchDF._jdf.s

[86conventional]

I'm trying to perform a merge inside a streaming foreachbatch using the command: microBatchDF._jdf.sparkSession().sql(self.merge_query)Streaming

[Kaniz_Fatma]

Hi @86conventional, For clusters with Databricks Runtime version 10.5 and above, you can access the local Spark session within the foreachBatch method.

If you encounter any further issues, feel free to ask for more assistance! 😊

For additional references, you can check out these resources:

• Medium Article on Complex Operations with Spark Structured Streaming
• Stack Overflow Discussion on Using Delta Tables with foreachBatch
• Stack Overflow Question about Removing Duplicate Records with foreachBatch