https://community.databricks.com/t5/data-engineering/auto-loader/m-p/62560#M31999 <P><a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/100565">@BhaveshPatel</a>&nbsp;</P><P>Three things that you can do:</P><P>- Move the files to the separate folder,<BR />- Use a filter on metadata fields to filter out the unnecessary files,<BR />- Use a&nbsp;<A href="https://spark.apache.org/docs/latest/sql-data-sources-generic-options.html#path-glob-filter" target="_self">pathGlobFilter</A>&nbsp;to filter only on the files you need</P> Mon, 04 Mar 2024 08:59:11 GMT daniel_sahal 2024-03-04T08:59:11Z https://community.databricks.com/t5/data-engineering/auto-loader/m-p/62285#M31941 <P>Suppose I have 1000's of historical .csv files stored from Jan, 2022 in a folder of my azure blob storage container. I want to use auto loader to read files beginning only on 1st, Oct, 2023 and ignoring all the files before this date to build a pipeline to read multiple files that are ingested daily.<BR />Is this possible using Auto loader, if yes, how?<BR />0 responses . Be the first to respond</P> Thu, 29 Feb 2024 02:08:01 GMT https://community.databricks.com/t5/data-engineering/auto-loader/m-p/62285#M31941 BhaveshPatel 2024-02-29T02:08:01Z https://community.databricks.com/t5/data-engineering/auto-loader/m-p/62560#M31999 <P><a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/100565">@BhaveshPatel</a>&nbsp;</P><P>Three things that you can do:</P><P>- Move the files to the separate folder,<BR />- Use a filter on metadata fields to filter out the unnecessary files,<BR />- Use a&nbsp;<A href="https://spark.apache.org/docs/latest/sql-data-sources-generic-options.html#path-glob-filter" target="_self">pathGlobFilter</A>&nbsp;to filter only on the files you need</P> Mon, 04 Mar 2024 08:59:11 GMT https://community.databricks.com/t5/data-engineering/auto-loader/m-p/62560#M31999 daniel_sahal 2024-03-04T08:59:11Z