https://community.databricks.com/t5/data-engineering/autoloader-process-multiple-files/m-p/16998#M11079 <P>I need to process files of different schema coming to different folders in ADLS using Autoloader. Do I need to start a separate read stream for each file type / folder or can this be handled using a single stream ?</P><P>When I tried using a single stream, multiple files gets clubbed in the same read and its quite difficult to process the files as the schema is different.</P> Tue, 13 Dec 2022 15:27:13 GMT avenu 2022-12-13T15:27:13Z https://community.databricks.com/t5/data-engineering/autoloader-process-multiple-files/m-p/16998#M11079 <P>I need to process files of different schema coming to different folders in ADLS using Autoloader. Do I need to start a separate read stream for each file type / folder or can this be handled using a single stream ?</P><P>When I tried using a single stream, multiple files gets clubbed in the same read and its quite difficult to process the files as the schema is different.</P> Tue, 13 Dec 2022 15:27:13 GMT https://community.databricks.com/t5/data-engineering/autoloader-process-multiple-files/m-p/16998#M11079 avenu 2022-12-13T15:27:13Z https://community.databricks.com/t5/data-engineering/autoloader-process-multiple-files/m-p/16999#M11080 <P>As you are talking about different schemas ,perhaps schemaevolutionmode, infercolumntypes, or schemahints may help?? Check out this- 32min onward - <A href="https://youtu.be/8a38Fv9cpd8" alt="https://youtu.be/8a38Fv9cpd8" target="_blank">https://youtu.be/8a38Fv9cpd8</A> </P><P>​</P><P>Hope it helps, do let know how you solve it if you can.​</P> Thu, 15 Dec 2022 14:50:42 GMT https://community.databricks.com/t5/data-engineering/autoloader-process-multiple-files/m-p/16999#M11080 Wassim 2022-12-15T14:50:42Z