https://community.databricks.com/t5/data-engineering/autoloader-ingestion-same-top-level-directory-different-files/m-p/66049#M33000 <P>Read all the files using auto loader and add an additional column as follows:</P><PRE>.withColumn("filePath",input_file_name())</PRE><P>Now that you've file name, you can split the data frame as per your requirement and ingest data into different tables.</P> Thu, 11 Apr 2024 06:44:00 GMT AmanSehgal 2024-04-11T06:44:00Z https://community.databricks.com/t5/data-engineering/autoloader-ingestion-same-top-level-directory-different-files/m-p/66029#M32989 <P>Hello,&nbsp;</P><P>Currently I have files landing in a storage account. They are all located in subfolders of a common directory. Some subdirectories may contain files, others may not. Each file name is unique and corresponds to a unique table as well. No two files are updating the same table. Is it possible to use autoloader and cloudfiles in a such a way that it can be given the path to a main directory as input, propagate through it, and process the data per file?</P> Wed, 10 Apr 2024 17:40:44 GMT https://community.databricks.com/t5/data-engineering/autoloader-ingestion-same-top-level-directory-different-files/m-p/66029#M32989 anonymous_567 2024-04-10T17:40:44Z https://community.databricks.com/t5/data-engineering/autoloader-ingestion-same-top-level-directory-different-files/m-p/66049#M33000 <P>Read all the files using auto loader and add an additional column as follows:</P><PRE>.withColumn("filePath",input_file_name())</PRE><P>Now that you've file name, you can split the data frame as per your requirement and ingest data into different tables.</P> Thu, 11 Apr 2024 06:44:00 GMT https://community.databricks.com/t5/data-engineering/autoloader-ingestion-same-top-level-directory-different-files/m-p/66049#M33000 AmanSehgal 2024-04-11T06:44:00Z