https://community.databricks.com/t5/data-engineering/is-autoloader-suitable-to-load-full-dumps/m-p/55005#M30216 <P>Hi,</P><P>I recently completed the fundamentals &amp; advanced data engineer exam, yet I've got a question about Autoloader. Please don't go too hard on me, since I lack practical experience at this point in time <span class="lia-unicode-emoji" title=":winking_face:">😉</span></P><P>Docs say this is incremental ingestion, so it's easy to load new files that contain all-new records that go into the stream. There's also the option to allow overwriting of files. What if the files provided by a source system are</P><P>A) full dumps which contain ALL records currently present in the system (missing records were deleted) so the loader needs to check for new, changed or missing records</P><P>B) delta, only new or changed records (deletes must be flagged)</P><P>Is Autoloader/COPY INTO still a good fit? Perhaps using a MERGE logic?</P><P>Thanks</P><P>Roger</P> Sun, 10 Dec 2023 19:54:41 GMT quakenbush 2023-12-10T19:54:41Z https://community.databricks.com/t5/data-engineering/is-autoloader-suitable-to-load-full-dumps/m-p/55005#M30216 <P>Hi,</P><P>I recently completed the fundamentals &amp; advanced data engineer exam, yet I've got a question about Autoloader. Please don't go too hard on me, since I lack practical experience at this point in time <span class="lia-unicode-emoji" title=":winking_face:">😉</span></P><P>Docs say this is incremental ingestion, so it's easy to load new files that contain all-new records that go into the stream. There's also the option to allow overwriting of files. What if the files provided by a source system are</P><P>A) full dumps which contain ALL records currently present in the system (missing records were deleted) so the loader needs to check for new, changed or missing records</P><P>B) delta, only new or changed records (deletes must be flagged)</P><P>Is Autoloader/COPY INTO still a good fit? Perhaps using a MERGE logic?</P><P>Thanks</P><P>Roger</P> Sun, 10 Dec 2023 19:54:41 GMT https://community.databricks.com/t5/data-engineering/is-autoloader-suitable-to-load-full-dumps/m-p/55005#M30216 quakenbush 2023-12-10T19:54:41Z