Get the list of loaded files from Autoloader

[herry]

Hello,

We can use Autoloader to track the files that have been loaded from S3 bucket or not. My question about Autoloader: is there a way to read the Autoloader database to get the list of files that have been loaded?

I can easily do this in AWS Glue job bookmark, but I'm not aware on how to do this in Databricks Autoloader.

[Hubert-Dudek · Accepted solution]

  .load("path")
  .withColumn("filePath",input_file_name())

than you can for example insert filePath to your stream sink and than get distinct value from there or use forEatch / forEatchBatch and for example insert it into spark sql table

[herry]

Thank you! This works for me 🙏

[kumar_ravi]

More efficient way

SELECT * FROM cloud_files_state('path/to/checkpoint');

[Anonymous]

@Herry Ramli​ - Would you be happy to mark Hubert's answer as best so that other members can find the solution more easily?

Thanks!