https://community.databricks.com/t5/data-engineering/autoloader-start-and-end-date-for-ingestion/m-p/45523#M27914 <P>I have been searching for a way to set up backfilling using autoloader with an option to set a "start_date" or "end_date". I am working on ingesting a massive file system but I don't want to ingest everything from the beginning. I have a start date that I want to perform the first big ingestion to populate the most recent data into my database and then over time slowly backfill the older data. Is this functionality currently in the autoloader settings, and if not, any suggestions on how to approach this issue?</P> Thu, 21 Sep 2023 13:04:48 GMT kmorton 2023-09-21T13:04:48Z https://community.databricks.com/t5/data-engineering/autoloader-start-and-end-date-for-ingestion/m-p/45523#M27914 <P>I have been searching for a way to set up backfilling using autoloader with an option to set a "start_date" or "end_date". I am working on ingesting a massive file system but I don't want to ingest everything from the beginning. I have a start date that I want to perform the first big ingestion to populate the most recent data into my database and then over time slowly backfill the older data. Is this functionality currently in the autoloader settings, and if not, any suggestions on how to approach this issue?</P> Thu, 21 Sep 2023 13:04:48 GMT https://community.databricks.com/t5/data-engineering/autoloader-start-and-end-date-for-ingestion/m-p/45523#M27914 kmorton 2023-09-21T13:04:48Z https://community.databricks.com/t5/data-engineering/autoloader-start-and-end-date-for-ingestion/m-p/101279#M40614 <P>If the files have already been loaded by autoloader (like same name and path), this can be tricky.</P> <P>I recommend starting a separate autoloader stream and specifying filters on it to match your start and end dates. If you'd instead like to rely on the modification timestamps of the files, you can use the modifiedBefore and modifiedAfter options.</P> Fri, 06 Dec 2024 20:42:23 GMT https://community.databricks.com/t5/data-engineering/autoloader-start-and-end-date-for-ingestion/m-p/101279#M40614 cgrant 2024-12-06T20:42:23Z