topic Re: Streaming Failure on Full Refresh Tables while using Serverless in Data Engineering

Streaming Failure on Full Refresh Tables while using Serverless

EAnthemNHC1 — Thu, 31 Jul 2025 12:30:13 GMT

On the afternoon of the 2025-07-30 my team began to experience issues with pipeline tasks that were set to full refresh and full refresh only. These pipelines were defined to use serverless, and the only way we were able to get them back online was to convert back to classic compute. I was unable to find much information about the error - has anyone else ran in to this, or have any idea what is occurring?

Error Message:
org.apache.spark.sql.streaming.StreamingQueryException: [STREAM_FAILED] Query terminated with exception: Dynamic admission control isn't available for batch 0 SQLSTATE: XXKST

To repeat, this only affected 'full refresh' pipeline tasks that were set to run in a serverless environment.

Re: Streaming Failure on Full Refresh Tables while using Serverless

nayan_wylde — Thu, 31 Jul 2025 15:09:24 GMT

Is your workspace in Private endpoint. If yes please check the NCC configuration and check if the private endpoint rule was created for each of the source and destination storages that you are using in this pipeline.

https://learn.microsoft.com/en-us/azure/databricks/security/network/serverless-network-security/serverless-private-link

Re: Streaming Failure on Full Refresh Tables while using Serverless

EAnthemNHC1 — Fri, 08 Aug 2025 15:42:34 GMT

Thanks for the reply - after consulting with our Databricks rep we determined it was a bug released by Databricks with a recent update to serverless. The Databricks team has resolved the issue and we have switched back to serverless.