I have an S3 bucket that has continuous data being written into it. My script reads these files, parses them and then appends into a delta table. The data backs to 2022 with millions of files which are stored using partitions based on year/month/dayO...
