Support Request: Issue Running Multiple Ingestion Gateway Concurrently

[Surya-Prathap]

Hi, we are ingesting data using Databricks Lake flow SQL connector from two different SQL Server databases hosted on separate servers. As part of the setup:

• We created two separate Ingestion Gateways.
• We created two separate ingestion pipelines.
• Both pipelines run successfully when executed individually, but they fail when executed concurrently.

When attempting to run both Ingestion Gateways at the same time, we encounter the following error:

Error Details:

com.databricks.pipelines.common.errors.deployment.DeploymentException:

[DLT ERROR CODE: QUOTA_EXCEEDED_EXCEPTION]

Cannot start update 'c04911e6-674f-40b7-93ba-df4c13e297ce' because the limit for active pipelines of type 'INGESTION_GATEWAY' has been reached

Can anyone please advise on how we can handle multiple concurrent ingestion streams within the current quota or any recommended approach to overcome this limitation?

[Louis_Frolio]

Greetings @Surya-Prathap , what you’re seeing lines up with a workspace-level quota on active Lakeflow Declarative Pipelines of type INGESTION_GATEWAY. In plain terms, you’ve hit the per-type concurrency cap, so the second gateway just isn’t allowed to start.

The PIPELINE_TYPE_QUOTA_EXCEEDED error is raised when the workspace has already reached its quota for active pipelines of a given type (in this case, INGESTION_GATEWAY).

Under the hood this is enforced by Gatekeeper via something like max_active_pipelines_per_type. If that’s effectively set to 1 (for example due to a trust-tier misclassification or a temporary config), only a single gateway pipeline can run at a time in that workspace.

In a normal, non-trial workspace, the default limit for concurrently running Lakeflow pipelines is much higher (typically in the hundreds) at the workspace level. So being capped at a single concurrent gateway is almost always a sign of a workspace-level quota or config issue, not a fundamental limitation of Lakeflow Connect.

Both of your gateways are configured as continuous pipelines by design, so once the first one is running and holding that slot, the second one hits the quota wall when it tries to start.

You are either running Free Edition or on a Trial Edition of Databricks. This is the expected behavior.

Hope this helps, Louis.

[Raman_Unifeye]

I would be using one Ingestion gateway and one ingestion pipeline (pointing to the same Ingestion Gateway) with 2 concurrent tasks (one for each ingestion). That will only counting as one active pipeline towards the quota.

Try it and share the results pls.

[HarishPrasath25]

@Raman_Unifeye , we have 2 databases and both are hosted in different server, is it possible to point both into one gateway?

[HarishPrasath25]

Hi @Louis_Frolio , I’ve successfully ingested one SQL database using the Lakeflow SQL connector. As part of the setup, I created an ingestion pipeline along with a gateway, and it is working as expected - when I run or re-run the pipeline, it picks up the new records correctly.

Now, I’m trying to set up ingestion for another SQL database hosted on a different server. This requires creating a new ingestion pipeline and a new gateway. However, the new gateway fails to start and throws the same quota exceeded error.

Could you please advise on how we can resolve this issue?
For context, we are using the Databricks Premium tier.