topic Get the list of loaded files from Autoloader in Data Engineering

Get the list of loaded files from Autoloader

herry — Mon, 06 Dec 2021 11:12:04 GMT

Hello,

We can use Autoloader to track the files that have been loaded from S3 bucket or not. My question about Autoloader: is there a way to read the Autoloader database to get the list of files that have been loaded?

I can easily do this in AWS Glue job bookmark, but I'm not aware on how to do this in Databricks Autoloader.

Re: Get the list of loaded files from Autoloader

Hubert-Dudek — Mon, 06 Dec 2021 11:25:36 GMT

  .load("path")
  .withColumn("filePath",input_file_name())

than you can for example insert filePath to your stream sink and than get distinct value from there or use forEatch / forEatchBatch and for example insert it into spark sql table

Re: Get the list of loaded files from Autoloader

herry — Thu, 09 Dec 2021 15:55:31 GMT

Thank you! This works for me 🙏

Re: Get the list of loaded files from Autoloader

Anonymous — Thu, 09 Dec 2021 16:12:42 GMT

@Herry Ramli - Would you be happy to mark Hubert's answer as best so that other members can find the solution more easily?

Thanks!

Re: Get the list of loaded files from Autoloader

kumar_ravi — Thu, 01 Aug 2024 21:03:12 GMT

More efficient way

SELECT * FROM cloud_files_state('path/to/checkpoint');