AutoLoader - process multiple files

[avenu]

I need to process files of different schema coming to different folders in ADLS using Autoloader. Do I need to start a separate read stream for each file type / folder or can this be handled using a single stream ?

When I tried using a single stream, multiple files gets clubbed in the same read and its quite difficult to process the files as the schema is different.

[Wassim]

As you are talking about different schemas ,perhaps schemaevolutionmode, infercolumntypes, or schemahints may help?? Check out this- 32min onward - https://youtu.be/8a38Fv9cpd8

Hope it helps, do let know how you solve it if you can.​