Initial loads can be a total nightmare. Imagine that every day you ingest 1 TB of data, but for the initial load, you need to ingest the last 5 years in a single pass. Roughly, thatโs 1 TB ร 365 days ร 5 years = 1825 TB of data. The new row_filter setting in Lakeflow Connect helps to handle it. #databricks
https://databrickster.medium.com/the-nightmare-of-initial-load-and-how-to-tame-it-9c81c2a4fbf7
https://www.sunnydata.ai/blog/initial-data-load-best-practices-databricks
