Spark Job fails with No plan for OptimizedForeachBatchFastpath

[SravanThotakura]

Hi Team,

I am trying to run a job on Databricks cluster 14.3LTS which streams data from parquet to custom sink. I am facing below error. The same code used to work a month back, however i am facing this issue recently.

org.apache.spark.SparkException: [INTERNAL_ERROR] The Spark SQL phase planning failed with an internal error. You hit a bug in Spark or the Spark plugins you use. Please, report this bug to the corresponding communities or vendors, and provide the full stack trace. SQLSTATE: XX000

Caused by: java.lang.AssertionError: assertion failed: No plan for OptimizedForeachBatchFastpath

[Walter_C · Accepted solution]

The error you are encountering, java.lang.AssertionError: assertion failed: No plan for OptimizedForeachBatchFastpath, appears to be related to a known issue with the OptimizedForeachBatchFastpath node in Databricks Runtime (DBR) 14.3 LTS. This issue can occur when using the foreachBatch sink in streaming jobs, particularly when Photon is enabled.

Here are some steps you can take to mitigate this issue:

1. Disable the Optimized Fast Path for foreachBatch:

   You can disable the optimized fast path for foreachBatch by setting the following Spark configuration:

   spark.conf.set("spark.databricks.streaming.forEachBatch.optimized.fastPath.enabled", "false")

   This can be done at the cluster level in the cluster configuration under Advanced options -> Spark, or within your notebook.

2. Disable Photon: If the issue persists, you can try disabling Photon to see if it resolves the problem. Photon can be disabled in the cluster configuration settings.

[SravanThotakura]

Thanks. Disabling Photon acceleration on the cluster fixed the issue

[Walter_C]

Great to hear the issue got fixed.