The core answer is: Many users encounter failures in structured streaming pipelines when switching from Databricks normal (classic) compute to Serverless, especially when using read streams on Unity Catalog Delta tables with Change Data Feed (CDF) enabled and .foreachBatch logic. This is typically not a code issue, but relates to current limitations and requirements of Databricks Serverless compute and Unity Catalog streaming support.

Why Serverless Fails with This Pattern

Databricks Serverless SQL warehouses have features and restrictions that differ from regular clusters. Common issues include:

• Streaming is not fully supported on Serverless warehouses: As of DBR 13.x–15.x, many structured streaming operations do not work with Serverless. This includes readStream from Unity Catalog tables, and especially Change Data Feed (CDF), which often causes failures on Serverless.

• Python streaming jobs depend on underlying compute: Serverless is primarily aimed at SQL, and Spark (Python) streaming jobs may fail or lack full feature parity.

• Delta table streaming and CDF with foreachBatch: ForeachBatch streaming logic requires certain capabilities in cluster management (e.g., persistent storage, job orchestration), which normal clusters provide but Serverless does not.

• Permissions and Unity Catalog: Serverless compute uses different managed identities, and lack of direct user control over runtime can lead to authentication or permission problems when accessing Unity Catalog tables for streaming.

Official Limitations and Known Issues

Here's a table summarizing streaming capabilities across environments:

Troubleshooting and Solution Strategies

• Use Jobs or Classic Clusters: For Python-based streaming, Databricks recommends running jobs on regular clusters, not Serverless.

• Check official documentation: Review Databricks docs for latest Serverless and Unity Catalog streaming support. Look for "supported operations" tables for your DBR version.

• Alternative: SQL Streaming Warehouses: If you must use Serverless, ensure your logic fits within SQL streaming pipelines, and use supported notebook/SQL features only (no Python, no foreachBatch).

• Permissions: If moving to a supported pipeline, ensure the Serverless warehouse has read/write permissions to Unity Catalog tables and checkpoint path.

Known Workarounds

• For Python/Spark streaming with CDF/foreachBatch, stick with regular compute.

• Consider batch ETL patterns or use SQL streaming if you must run Serverless.

References

• Databricks Unity Catalog streaming limitations, especially with Serverless compute.

• Discussions of similar failures on Databricks forums and StackOverflow, identifying feature gaps for Serverless.

In summary: The error arises because Databricks Serverless compute does not currently support Python-based streaming jobs on Unity Catalog tables with CDF and foreachBatch. To resolve, switch back to regular clusters for streaming, or adjust your workflow to use supported SQL streaming pipelines on Serverless if possible.