https://community.databricks.com/t5/machine-learning/when-should-you-use-the-directory-listing-vs-file-notification/m-p/5064#M234 <P>@Bennett Lambert​&nbsp;:</P><P>The choice between using "file notification" vs "directory listing" for the autoloader in Delta Live Tables depends on your specific use case and requirements. Here are some general guidelines:</P><OL><LI>Use file notification if you need real-time ingestion: File notification uses event-based triggers to detect new files in a source storage location, which allows for real-time ingestion as soon as new files are added.</LI><LI>Use directory listing if you need to control the ingestion frequency: Directory listing periodically scans the source storage location for new files, which allows you to control the frequency of ingestion. This can be useful if you need to limit the number of ingested files or control the timing of ingestion.</LI><LI>Use file notification for small files: File notification is more efficient for small files because it avoids scanning the entire directory for changes.</LI><LI>Use directory listing for large files: Directory listing is more efficient for large files because it can reduce the overhead of scanning and processing each file individually.</LI></OL><P>In summary, if you need real-time ingestion and have a large number of small files, use file notification. If you need to control the ingestion frequency or have a small number of large files, use directory listing.</P> Sat, 13 May 2023 16:58:07 GMT Anonymous 2023-05-13T16:58:07Z https://community.databricks.com/t5/machine-learning/when-should-you-use-the-directory-listing-vs-file-notification/m-p/5063#M233 <P>We are using Delta Live Tables for running ingestion pipelines and have come across the two options for the autoloader "file notification" vs "directory listing" this is reflected in the option cloudFiles.useIncrementalListing. We are wondering what the best practices are around which of these to use for the autoloader and when we should use one vs the other?</P> Fri, 28 Apr 2023 08:37:12 GMT https://community.databricks.com/t5/machine-learning/when-should-you-use-the-directory-listing-vs-file-notification/m-p/5063#M233 BenLambert 2023-04-28T08:37:12Z https://community.databricks.com/t5/machine-learning/when-should-you-use-the-directory-listing-vs-file-notification/m-p/5064#M234 <P>@Bennett Lambert​&nbsp;:</P><P>The choice between using "file notification" vs "directory listing" for the autoloader in Delta Live Tables depends on your specific use case and requirements. Here are some general guidelines:</P><OL><LI>Use file notification if you need real-time ingestion: File notification uses event-based triggers to detect new files in a source storage location, which allows for real-time ingestion as soon as new files are added.</LI><LI>Use directory listing if you need to control the ingestion frequency: Directory listing periodically scans the source storage location for new files, which allows you to control the frequency of ingestion. This can be useful if you need to limit the number of ingested files or control the timing of ingestion.</LI><LI>Use file notification for small files: File notification is more efficient for small files because it avoids scanning the entire directory for changes.</LI><LI>Use directory listing for large files: Directory listing is more efficient for large files because it can reduce the overhead of scanning and processing each file individually.</LI></OL><P>In summary, if you need real-time ingestion and have a large number of small files, use file notification. If you need to control the ingestion frequency or have a small number of large files, use directory listing.</P> Sat, 13 May 2023 16:58:07 GMT https://community.databricks.com/t5/machine-learning/when-should-you-use-the-directory-listing-vs-file-notification/m-p/5064#M234 Anonymous 2023-05-13T16:58:07Z