Databricks Community

esalohs · 11-21-2023

I have an s3 bucket with a couple of subdirectories/partitions like s3a://Bucket/dir1/ and s3a://Bucket/dir2/. There is currently in the millions of files sitting in bucket in the various subdirectories/partitions. I'm getting new data in near real t...

esalohs · 12-13-2023

would you happen to still have your code/outline/options you set handy? That would be incredible if you did @akul

esalohs · 12-13-2023

hey @kulkpd - unfortunately I don't have file notifications enabled on my cluster. i did the initial scan and it took about 5 days as there were 30M files sitting in there. However, when I try and get the 'incremental files' it's still taking a fair ...

esalohs · 11-23-2023

Thanks @Retired_mod ! So I don't need to do an initial one time scan on the entire bucket/directory to commit the existing files to rocks? I can just use this to grab the incremental loads every 15 min? Thanks again for the help 

Databricks Community

User Stats

User Activity

Databricks Autoloader - list only new files in an s3 bucket/directory

Re: Databricks Autoloader - list only new files in an s3 bucket/directory

Re: Databricks Autoloader - list only new files in an s3 bucket/directory

Re: Databricks Autoloader - list only new files in an s3 bucket/directory