Hi Team,
I would like to understand if there is a metadata table for the autoloader in Databricks that captures information about file arrival and processing.
The reason we are experiencing data issues is because our table A receives hundreds of files that are processed by an autoloader,
and in some scenarios, we have noticed that old files are processed after new files, possibly due to a problem in the source system.
However, if we have clear details about autoloder metadata, it will be easier to identify the root cause analysis.
could you please share the best practices for organizing data in a storage location that an autoloader can effectively process?
