topic Autoloader file latency in Get Started Discussions

Autoloader file latency

Phani1 — Tue, 30 Jan 2024 16:27:41 GMT

Hi Team,

I would like to understand if there is a metadata table for the autoloader in Databricks that captures information about file arrival and processing.

The reason we are experiencing data issues is because our table A receives hundreds of files that are processed by an autoloader,
and in some scenarios, we have noticed that old files are processed after new files, possibly due to a problem in the source system.
However, if we have clear details about autoloder metadata, it will be easier to identify the root cause analysis.

could you please share the best practices for organizing data in a storage location that an autoloader can effectively process?

Re: Autoloader file latency

jose_gonzalez — Wed, 31 Jan 2024 00:06:13 GMT

Are you using file listing or file notification for auto loader?

Re: Autoloader file latency

Phani1 — Wed, 31 Jan 2024 04:58:29 GMT

we are using the default.

Re: Autoloader file latency

jose_gonzalez — Wed, 31 Jan 2024 19:10:53 GMT

Check with

cloud_files_state() API

You can find examples here https://docs.databricks.com/en/ingestion/auto-loader/production.html#querying-files-discovered-by-auto-loader