Autoloader ingestion same top level directory different files corresponding to different tables
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-10-2024 10:40 AM
Hello,
Currently I have files landing in a storage account. They are all located in subfolders of a common directory. Some subdirectories may contain files, others may not. Each file name is unique and corresponds to a unique table as well. No two files are updating the same table. Is it possible to use autoloader and cloudfiles in a such a way that it can be given the path to a main directory as input, propagate through it, and process the data per file?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-10-2024 11:44 PM
Read all the files using auto loader and add an additional column as follows:
.withColumn("filePath",input_file_name())Now that you've file name, you can split the data frame as per your requirement and ingest data into different tables.
