Hi @yit337,

For Lakeflow Connect managed ingestion pipelines, the behaviour you’re seeing is expected. As soon as you use ingestion_definition in your pipeline (i.e., a managed ingestion pipeline), Databricks forces that pipeline to run on serverless compute. Serverless pipelines do not allow a clusters block. If you include cluster settings, you get the error you are seeing.

You can refer to this documentation page.

 If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.

Yes — Lakeflow Connect ingestion pipelines always run on Serverless compute. 

 

Came across this as I hit the same error. The documentation suggests that using classic compute is supported, albeit in beta  - https://learn.microsoft.com/en-gb/azure/databricks/ingestion/lakeflow-connect/query-based-overview#c...

Hi @SamuelGTurner,

Yes, it was released to beta in April following the original post. Please note that classic compute can only be used with the API. Databricks still recommends using Serverless.