Is it required to run Lakeflow Connect on Serverless?

[yit337]

As the subject states, my question is:

Is it required to run the Ingestion Pipeline in Lakeflow Connect on Serverless compute? Cause I try to define my own cluster in the DAB, but it raises an error:

`Error: cannot create pipeline: You cannot provide cluster settings when using serverless compute.` 

Even though my serverless flag is set to false.

[Ashwin_DSA]

Hi @yit337,

For Lakeflow Connect managed ingestion pipelines, the behaviour you’re seeing is expected. As soon as you use ingestion_definition in your pipeline (i.e., a managed ingestion pipeline), Databricks forces that pipeline to run on serverless compute. Serverless pipelines do not allow a clusters block. If you include cluster settings, you get the error you are seeing.

You can refer to this documentation page.

 If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.

[saurabh18cs]

Yes — Lakeflow Connect ingestion pipelines always run on Serverless compute. 

Databricks overrides your compute config and switches back to serverless,
because the ingestion connector requires it.