Hi Team,

I am doing ingestion of 10 tables from Azure SQL through Lakeflow connect. I have created gateway and ingestion pipelines using databricks SDK. I am starting ingestion pipeline only when gateway is in Running status with resources. 
I observed that in first load only few tables are ingesting and partial load for one big table (30 ml rows). when I checked the gateway event log, snapshot was started only when ingestion pipeline triggered but ingestion is not waiting for snapshot to gets completed and load the available data from cdc_stage table to delta tables.
how can I overcome this issue of partial load in first run and loads all the historical data available in source sql database? any configuration while creating gateway or any workaround?
P.S. : I also added 15 mins sleep time before I trigger ingestion pipeline to test but snapshot started only when ingestion started.

#lakeflow #ingestion #lakeflowConnect