Autoloader file latency

[Phani1]

Hi Team,

I would like to understand if there is a metadata table for the autoloader in Databricks that captures information about file arrival and processing.

The reason we are experiencing data issues is because our table A receives hundreds of files that are processed by an autoloader,
and in some scenarios, we have noticed that old files are processed after new files, possibly due to a problem in the source system.
However, if we have clear details about autoloder metadata, it will be easier to identify the root cause analysis.

could you please share the best practices for organizing data in a storage location that an autoloader can effectively process?

[jose_gonzalez]

Are you using file listing or file notification for auto loader? 

[Phani1]

we are using the default.

[jose_gonzalez]

Check with 

cloud_files_state() API

You can find examples here https://docs.databricks.com/en/ingestion/auto-loader/production.html#querying-files-discovered-by-au...

[Kaniz_Fatma]

Hey there! Thanks a bunch for being part of our awesome community! 🎉 

We love having you around and appreciate all your questions. Take a moment to check out the responses – you'll find some great info. Your input is valuable, so pick the best solution for you. And remember, if you ever need more help , we're here for you! 

Keep being awesome! 😊🚀