- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-06-2025 01:09 AM
Hi @ChristianRRL ,
In your case, it's perfectly fine to load data directly into Delta tables (and skip any “extra” steps). However, let me share a few examples of how using Auto Loader has made our lives significantly easier on various projects.
For instance, in one project, IoT devices were sending data to a Data Lake. Imagine millions of files landing on your storage account. Auto Loader automatically detects which new files have arrived since the last load (and gives you out-of-the-box fault tolerance and exactly-once semantics).
Now, sometimes due to issues with the IoT devices, we would receive duplicate data.
That's why we made sure the process loading data into the Silver layer was idempotent (using a MERGE statement).
One more thing - check whether your API has any data retention limitations. For one client, we were integrating data from Google Search Console. It turned out that Google Search Console only retains data for the last 16 months. So in that case, we had to land the data first in a landing path in order to build a history longer than 16 months. Otherwise, if we ever needed to reprocess the data, we would lose historical records (since anything older than 16 months would no longer be available via the API).