Databricks Community

Hey everyone! I am trying to switch from a batch processing job to a autoloader (directory listing mode) on my s3 bucket that has millions of files. I am using modifiedAfter to create my initial checkpoint for the autoloader and want to speed up the ...

I have an S3 bucket that has continuous data being written into it. My script reads these files, parses them and then appends into a delta table. The data backs to 2022 with millions of files which are stored using partitions based on year/month/dayO...