Issue with the SQL Server ingestion in Databricks Lakflow- Gateway Stuck at "Starting"

[AshwinGujarathi]

Context: I am working on a POC to explore Databricks Lakeflow ingestion feature. I have tried creating Ingestion pipeline with salesforce connector successfully and able to load the data into Delta table. However when I am trying to create ingestion pipeline with sql server connector I am facing below issue.

Issue Description:  I am facing an issue while trying to ingest data from Azure SQL Server using LakeFlow in my Databricks workspace. The ingestion pipeline is getting stuck at the "Gateway starting - Waiting for gateway compute resources.." step and not processing beyond that. Also I am not getting any error and pipeline is stuck at step 3 of the ingestion process. Please refer snapshot for reference.

I have created sample table in Azure SQL Database and able to access it using Azure Data Studio.

What I have already tried:  

1. Confirmed the SQL Server is online and reachable. 
2. Verified the firewall settings allow Azure services to access SQL Server.
3. Tried recreating the connection and ingestion process.
4. Ensured correct connection string and credentials are in place.

I would appreciate your help in identifying the root cause and resolving the issue.

Thank you,

Ashwin

[dbernstein_tp]

I am having a similar problem but in my case I get this error after about 20 minutes of the gateway cluster trying to start:

com.databricks.pipelines.common.errors.deployment.DeploymentException: Failed to launch pipeline cluster 0829-204342-h9tqov5v: Encountered Quota Exhaustion issue in your account:  databricks_error_message: The VM size you are specifying is not available. [details] QuotaExceeded: Operation could not be completed as it resul...
This error is likely due to a misconfiguration in the pipeline.
Check the pipeline cluster configuration and associated cluster policy.

The gateway is trying to create a cluster of type "Standard_F4s" and we have checked that we have sufficient quotas for this family.

[dbernstein_tp]

We were able to resolve the above issue by increasing the quotas for two separate compute families. One was the Standard F4s family, which apparently the worker uses and the other was the Standard EDV4 family which I guess is the driver node. Now we're having another issue but that is for a different topic...