Autoloader ingestion same top level directory different files corresponding to different tables

[anonymous_567]

Hello, 

Currently I have files landing in a storage account. They are all located in subfolders of a common directory. Some subdirectories may contain files, others may not. Each file name is unique and corresponds to a unique table as well. No two files are updating the same table. Is it possible to use autoloader and cloudfiles in a such a way that it can be given the path to a main directory as input, propagate through it, and process the data per file?

[AmanSehgal]

Read all the files using auto loader and add an additional column as follows:

.withColumn("filePath",input_file_name())

Now that you've file name, you can split the data frame as per your requirement and ingest data into different tables.