- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-13-2025 02:24 AM
Hi Team,
I am trying to run a job on Databricks cluster 14.3LTS which streams data from parquet to custom sink. I am facing below error. The same code used to work a month back, however i am facing this issue recently.
org.apache.spark.SparkException: [INTERNAL_ERROR] The Spark SQL phase planning failed with an internal error. You hit a bug in Spark or the Spark plugins you use. Please, report this bug to the corresponding communities or vendors, and provide the full stack trace. SQLSTATE: XX000
Caused by: java.lang.AssertionError: assertion failed: No plan for OptimizedForeachBatchFastpath
Accepted Solutions
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-13-2025 04:04 AM
The error you are encountering, java.lang.AssertionError: assertion failed: No plan for OptimizedForeachBatchFastpath
, appears to be related to a known issue with the OptimizedForeachBatchFastpath
node in Databricks Runtime (DBR) 14.3 LTS. This issue can occur when using the foreachBatch
sink in streaming jobs, particularly when Photon is enabled.
Here are some steps you can take to mitigate this issue:
-
Disable the Optimized Fast Path for
You can disable the optimized fast path forforeachBatch
:foreachBatch
by setting the following Spark configuration:spark.conf.set("spark.databricks.streaming.forEachBatch.optimized.fastPath.enabled", "false")
This can be done at the cluster level in the cluster configuration under Advanced options -> Spark, or within your notebook.
-
Disable Photon: If the issue persists, you can try disabling Photon to see if it resolves the problem. Photon can be disabled in the cluster configuration settings.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-13-2025 04:04 AM
The error you are encountering, java.lang.AssertionError: assertion failed: No plan for OptimizedForeachBatchFastpath
, appears to be related to a known issue with the OptimizedForeachBatchFastpath
node in Databricks Runtime (DBR) 14.3 LTS. This issue can occur when using the foreachBatch
sink in streaming jobs, particularly when Photon is enabled.
Here are some steps you can take to mitigate this issue:
-
Disable the Optimized Fast Path for
You can disable the optimized fast path forforeachBatch
:foreachBatch
by setting the following Spark configuration:spark.conf.set("spark.databricks.streaming.forEachBatch.optimized.fastPath.enabled", "false")
This can be done at the cluster level in the cluster configuration under Advanced options -> Spark, or within your notebook.
-
Disable Photon: If the issue persists, you can try disabling Photon to see if it resolves the problem. Photon can be disabled in the cluster configuration settings.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-13-2025 04:44 AM
Thanks. Disabling Photon acceleration on the cluster fixed the issue
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-13-2025 06:04 AM
Great to hear the issue got fixed.

