Thanks for the suggestion. I am not using structured streaming in the code. Not sure, if i can use `foreachBatch()` without it.

I have added a `try` block around the `foreach()` logic and added the `foreachPartition()` logic in an `except` block.

Also, the error resolved itself without any changes from our side. We are still awaiting response from Databricks support as to why the error intermittently appeared.