data_pulse
New Contributor III

@yit337 
I spent some time in reproducing this in a regular Databricks workspace using minimal Lakeflow Declarative Pipeline.

The pipeline contained only:
CREATE OR REFRESH MATERIALIZED VIEW serverless_test
AS
SELECT 1 AS id;

There were no expectations, constraints, CDC or other Advanced-only workload features.

Tested the following configurations:

ConfigurationResult
serverless: true, edition: PROFailed during pipeline creation while deploying the bundle itself
serverless: true, edition: ADVANCEDSucceeded
serverless: false, edition: PROSucceeded
serverless: true, edition omittedSucceeded

For serverless: true with edition: PRO, databricks bundle deploy failed before the pipeline or SQL executed with:

Error: cannot create pipeline: You must use the Advanced edition when using serverless compute.


This shows that the restriction is enforced at the pipeline configuration level and is not caused by expectations or another Advanced-only feature in the pipeline code.

So, at least in the workspace where I tested this, serverless pipelines require ADVANCED, and omitting edition causes Databricks to resolve the effective edition to ADVANCED.

The practical workarounds are:

  • Use serverless: true with edition: ADVANCED, or omit edition.
  • Use edition: PRO with non-serverless/classic pipeline compute.

This also appear to be a documentation gap on serverless pipeline constraint.

View solution in original post