Is Advanced edition required for Serverless pipeline?

[yit337]

I want to set Serverless pipeline with edition=PRO, but it raises the following error:  cannot update pipeline: You must use the Advanced edition when using serverless compute.

This is not specified anywhere in the documentation. It only says (see screenshot below) that data quality checks aren't allowed on other editions, but I don't have any checks in my pipeline. The error is vague. Either Serverless requires Advanced edition, or the error doesn't provide enough information on which Advanced features I'm using in the pipeline and should be changed to enable Pro.

[aayush_410]

If your pipeline is a plain Lakeflow Declarative Pipeline with no ingestion gateway involved, I couldn't find this constraint documented anywhere generally, which matches your frustration. One thing worth checking on your end: the Pipeline properties reference notes that the default value of edition is actually ADVANCED, not Core or Pro — so Databricks already treats Advanced as the baseline assumption for pipelines generally, which suggests this may be a recently tightened validation rule for serverless specifically (possibly to guarantee access to serverless-only capabilities like incremental refresh/vertical autoscaling that are gated behind Advanced-tier feature flags internally) rather than being tied to any specific feature you're using like expectations.

Practical next steps:

Check if your pipeline uses any AUTO CDC/APPLY CHANGES INTO flows — CDC constructs require at minimum Pro; if the error logic bundles "CDC + serverless" together it may be pushing you to Advanced even without expectations. Worth ruling out.
If cost is the concern for staying on Pro rather than Advanced: check current DBU pricing for your workspace, since Core/Pro/Advanced have historically been billed at different per-DBU rates — Advanced isn't necessarily prohibitively more expensive for a given workload, but worth confirming rather than assuming.
Since the error message is genuinely vague about which feature is triggering the requirement (as opposed to naming the specific unsupported construct, which is how the docs say edition-mismatch errors are supposed to behave — "you receive an error message explaining the reason for the error"), this looks like a legitimate product/docs gap worth filing directly with Databricks via the docs page feedback widget or a support ticket, quoting the exact error text and confirming you have no expectations in the pipeline. That's the fastest way to get either the docs corrected or the error message made specific.

[yit337]

Thank you for the extensive explanation! It helps at least to know that my use case is not explained in the documentation.

I can rule out CDC issues, because the same pipeline is executed on dev with Serverless=FALSE and Edition=PRO, and it works.

I need to have Serverless only on prod, so I set Serverless=TRUE and Edition=PRO, and the issues appear on prod only.

So the only difference is Serverless being enabled. Otherwise, if it was for specific feature, I'd assume that the pipeline wouldn't work on DEV as well.

[data_pulse · Accepted solution]

@yit337 
I spent some time in reproducing this in a regular Databricks workspace using minimal Lakeflow Declarative Pipeline.

The pipeline contained only:
CREATE OR REFRESH MATERIALIZED VIEW serverless_test
AS
SELECT 1 AS id;

There were no expectations, constraints, CDC or other Advanced-only workload features.

Tested the following configurations:

Configuration	Result
serverless: true, edition: PRO	Failed during pipeline creation while deploying the bundle itself
serverless: true, edition: ADVANCED	Succeeded
serverless: false, edition: PRO	Succeeded
serverless: true, edition omitted	Succeeded

For serverless: true with edition: PRO, databricks bundle deploy failed before the pipeline or SQL executed with:

Error: cannot create pipeline: You must use the Advanced edition when using serverless compute.

This shows that the restriction is enforced at the pipeline configuration level and is not caused by expectations or another Advanced-only feature in the pipeline code.

So, at least in the workspace where I tested this, serverless pipelines require ADVANCED, and omitting edition causes Databricks to resolve the effective edition to ADVANCED.

The practical workarounds are:

• Use serverless: true with edition: ADVANCED, or omit edition.
• Use edition: PRO with non-serverless/classic pipeline compute.

This also appear to be a documentation gap on serverless pipeline constraint.

[yit337]

Thank you for doing the comparison! As a conclusion, documentation gap..

[krishgarikipati]

Hi @yit337 

This is already mentioned in the Databricks documentation. PFB snippet for the reference. https://docs.databricks.com/gcp/en/ldp/properties?utm_source=chatgpt.com 

[yit337]

I think you misread something in my explanation. What I am explaining is that my pipeline does not use any of the advanced features, yet when I try to set Serverless=True and edition=Advanced, it raises errors. That's why I am asking whether Advanced is required for Serverless, which is not mentioned in the docs. In the snippet you've shared it explains that advanced features (such as data quality checks, CDC,...) require edition=Advanced, but I am not using any of those, yet the pipeline raises error when run on Serverless. 🙂

I have even attached the same screenshot you've shared above in my post as a proof that they are mentioning only that advanced is needed for some features, but not for running the pipeline in Serverless mode.

[Khasim_1]

Serverless DLT/Lakeflow pipelines currently mandate the "Advanced" edition.

Even if you are not using Data Quality expectations (the most famous "Advanced" feature), the Serverless orchestration itself relies on advanced infrastructure management, enhanced auto-scaling logic, and specialized monitoring that Databricks has bundled exclusively into the Advanced tier.

Why the error occurs

When you select Serverless as your compute mode, Databricks takes over the management of the underlying infrastructure. To ensure the reliability and performance of this managed service, the platform enforces the highest feature tier (Advanced).

Essentially, you are not paying the "Advanced" premium just for Data Quality checks; you are paying it for the fully managed, zero-ops infrastructure provided by Serverless.

[yit337]

I agree with everything you've said, we can justify the need of Advantage edition with Serverless, but they haven't made it clear in the documentation that Serverless compute requires Advanced edition. Even if you ask any of the LLMs, they will say that Serverless doesn't require the Advanced edition, so it is a lack of documentation for sure!