Context: I am working on a POC to explore Databricks Lakeflow ingestion feature. I have tried creating Ingestion pipeline with salesforce connector successfully and able to load the data into Delta table. However when I am trying to create ingestion pipeline with sql server connector I am facing below issue.
Issue Description: I am facing an issue while trying to ingest data from Azure SQL Server using LakeFlow in my Databricks workspace. The ingestion pipeline is getting stuck at the "Gateway starting - Waiting for gateway compute resources.." step and not processing beyond that. Also I am not getting any error and pipeline is stuck at step 3 of the ingestion process. Please refer snapshot for reference.
I have created sample table in Azure SQL Database and able to access it using Azure Data Studio.
What I have already tried:
- Confirmed the SQL Server is online and reachable.
- Verified the firewall settings allow Azure services to access SQL Server.
- Tried recreating the connection and ingestion process.
- Ensured correct connection string and credentials are in place.
I would appreciate your help in identifying the root cause and resolving the issue.
Thank you,
Ashwin
