My data source is going to generate csv files for multiple objects all into the same directory that I need to load from. The files will have names in the format along the lines of <objecttype>_YYYY_MM_DD_guid.csv.gz. Each objecttype will have it's own schema, and the goal is to load those files into objecttype tables.

I'm using Autoloader with cloudfiles to ingest these files. The objective is to have the process by dynamic and not have a static set of objecttypes, being able to adjust to new objecttypes being added. In my prototype, I'm performing an initial dbutils.fs.ls to get a file list and parse our the unique objecttypes. Then I loop through each objecttype, passing them to autoloader and globbing the objecttype out. But I feel this won't scale well as the number of files increases.

Is there a better method I can use to ingest these files and have them directed to the correct tables?