topic Re: Is it required to run Lakeflow Connect on Serverless? in Data Engineering

Is it required to run Lakeflow Connect on Serverless?

yit337 — Mon, 30 Mar 2026 09:15:09 GMT

As the subject states, my question is:

Is it required to run the Ingestion Pipeline in Lakeflow Connect on Serverless compute? Cause I try to define my own cluster in the DAB, but it raises an error:

`Error: cannot create pipeline: You cannot provide cluster settings when using serverless compute.`

Even though my serverless flag is set to false.

Re: Is it required to run Lakeflow Connect on Serverless?

Ashwin_DSA — Mon, 30 Mar 2026 10:59:22 GMT

Hi @yit337,

For Lakeflow Connect managed ingestion pipelines, the behaviour you’re seeing is expected. As soon as you use ingestion_definition in your pipeline (i.e., a managed ingestion pipeline), Databricks forces that pipeline to run on serverless compute. Serverless pipelines do not allow a clusters block. If you include cluster settings, you get the error you are seeing.

You can refer to this documentation page.

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.

Re: Is it required to run Lakeflow Connect on Serverless?

saurabh18cs — Mon, 30 Mar 2026 11:04:14 GMT

Yes — Lakeflow Connect ingestion pipelines always run on Serverless compute.

Databricks overrides your compute config and switches back to serverless,
because the ingestion connector requires it.