Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-23-2024 06:24 AM
What I wanted to achieve was a dynamic schema application based on what file was picked up.
So I implement 1 autoloader task to collect files from a specific path "source":
- source/employees/0001.csv
- source/holiday/0001.csv
If the path of the file was available I could then apply the relevant schema in runtime.
But autoloader may want to process both files and put them into the same dataframe?
Maybe this isn't the best usecase, I guess you would recommend to implement multiple tasks/checkpoints for the respective folders?