Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-25-2024 06:58 AM
your patterns with the date folders is the classic way of processing data in data lakes.
But with autoloader you can get rid of them.
Create a folder per table, f.e. /bronze/files/table1 and read that folder using autoloader.
Your filenames can be anything as long as the can be lexicographically sorted. I use <epoch>_filename...
Autoloader keeps track of what has been processed and what not.
So like that you have one autoloader per table.
Does that make sense?