Hey Nick,

Have you tried the SQL Server connector with Lakeflow Connect? This should provide native connection to your SQL server, potentially allowing for incremental updates and CDC setup. 

https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/sql-server-pipeline

I haven’t tried this connector before but it seems like a good first thing to try for your case. 

Kerem Durak