topic Spark Job fails with No plan for OptimizedForeachBatchFastpath in Administration & Architecture

Spark Job fails with No plan for OptimizedForeachBatchFastpath

SravanThotakura — Mon, 13 Jan 2025 10:24:36 GMT

Hi Team,

I am trying to run a job on Databricks cluster 14.3LTS which streams data from parquet to custom sink. I am facing below error. The same code used to work a month back, however i am facing this issue recently.

org.apache.spark.SparkException: [INTERNAL_ERROR] The Spark SQL phase planning failed with an internal error. You hit a bug in Spark or the Spark plugins you use. Please, report this bug to the corresponding communities or vendors, and provide the full stack trace. SQLSTATE: XX000

Caused by: java.lang.AssertionError: assertion failed: No plan for OptimizedForeachBatchFastpath

Re: Spark Job fails with No plan for OptimizedForeachBatchFastpath

Walter_C — Mon, 13 Jan 2025 12:04:06 GMT

The error you are encountering, java.lang.AssertionError: assertion failed: No plan for OptimizedForeachBatchFastpath, appears to be related to a known issue with the OptimizedForeachBatchFastpath node in Databricks Runtime (DBR) 14.3 LTS. This issue can occur when using the foreachBatch sink in streaming jobs, particularly when Photon is enabled.

Here are some steps you can take to mitigate this issue:

Disable the Optimized Fast Path for foreachBatch:
You can disable the optimized fast path for foreachBatch by setting the following Spark configuration:
```
spark.conf.set("spark.databricks.streaming.forEachBatch.optimized.fastPath.enabled", "false")
```
This can be done at the cluster level in the cluster configuration under Advanced options -> Spark, or within your notebook.
Disable Photon: If the issue persists, you can try disabling Photon to see if it resolves the problem. Photon can be disabled in the cluster configuration settings.

Re: Spark Job fails with No plan for OptimizedForeachBatchFastpath

SravanThotakura — Mon, 13 Jan 2025 12:44:26 GMT

Thanks. Disabling Photon acceleration on the cluster fixed the issue

Re: Spark Job fails with No plan for OptimizedForeachBatchFastpath

Walter_C — Mon, 13 Jan 2025 14:04:29 GMT

Great to hear the issue got fixed.

Re: Spark Job fails with No plan for OptimizedForeachBatchFastpath

Malthe — Tue, 23 Sep 2025 13:04:36 GMT

Disabling the setting worked for me on 16.4 (serverless):

spark.databricks.streaming.forEachBatch.optimized.fastPath.enabled=false

But you have to wonder if this is something Databricks are aware of and are looking to fix.