Managed File Events: Are reads from the file events cache independent per pipeline?

[raimundovidal]

We have two Databricks workspaces (staging and production) attached to the same Unity Catalog metastore. Both workspaces run DLT pipelines that use Auto Loader with cloudFiles.useManagedFileEvents = "true" to ingest from the same
external location (same S3 path).

Each pipeline has its own separate checkpoint location.

The documentation states that managed file events uses "a single file notification queue for all streams that process files from a given external location" and that streams discover new files by "reading directly from cache using
stored read position."

Our concern: If the staging pipeline runs first and reads new files from the file events cache, will the production pipeline still see those same files when it runs later? Or does one pipeline's read advance a shared cursor that
causes the other to miss files?

Specifically, we'd like to clarify:

1. Is the "stored read position" scoped per pipeline/stream (each pipeline independently tracks its own position in the cache) or is it shared across all consumers of the external location?
2. Is the file events cache designed to support multiple independent consumers reading the same file events without interference — similar to how Kafka consumer groups each maintain their own offset?

Our current understanding is that each pipeline maintains its own read position via its checkpoint, making this safe. But we couldn't find explicit documentation confirming this for cross-workspace, same-metastore scenarios.

Any clarification would be appreciated. Thanks!

[Ashwin_DSA · Accepted solution]

Hi @raimundovidal,

You’re safe to run both staging and production Lakeflow Spark Declarative Pipelines with cloudFiles.useManagedFileEvents = "true" against the same external location (same S3 path) and same Unity Catalog metastore, as long as each pipeline uses its own checkpoint location.

A few key points:

• When a stream first runs with managed file events, Auto Loader does a full directory listing of the load path to get current with the file events cache and then stores the read position in that stream’s checkpoint. Subsequent runs read from the cache using the stored read position rather than S3 directly.
• The checkpoint also records a unique stream ID, and the checkpointLocation is explicitly per stream.
• External locations (and their file events setup) are metastore-level objects and are, by default, accessible from all workspaces attached to that metastore. So two workspaces attached to the same metastore behave just like two streams in one workspace from the file-events perspective.

Putting that together... 

1. Is the “stored read position” scoped per pipeline/stream or shared?

 The stored read position is scoped per stream, not shared across all consumers of the external location. Each  Auto Loader stream (including each DLT pipeline) keeps its own position in the file events cache inside its own checkpoint. That position is obtained during the initial run and then reused on subsequent runs of that specific stream.

So in your scenario:

• If the staging pipeline runs first and advances its own read position, that does not move the read position for the production pipeline.
• When the production pipeline runs, it uses its own checkpoint and its own stored position in the cache, and will still see all new files (subject to the usual rules: e.g., it must run frequently enough that its stored position remains valid).
• There is no single “global cursor” for the external location that all streams share.

2. Can multiple independent consumers read the same file events without interfering (Kafka‑like semantics)?

Yes. That’s the intent.

A few implementation details that are relevant to your concerns:

• For an external location with file events enabled, Databricks configures one underlying queue per external location (not per stream).
• A control-plane service (CSMS) consumes from that queue and caches file metadata. Auto Loader streams do not read the queue directly; instead, they query this service using an internal listObjects‑style API, which returns objects plus a continuation token.
• Each stream keeps its own continuation token / read position (per external location) in its checkpoint. As a result, multiple streams can safely read from the same cache independently, very similar to distinct Kafka consumer groups each tracking their own offsets.

This design is explicitly used and supported when:

• Multiple Auto Loader streams ingest different subpaths under the same external location with managed file events.
• Customers run many concurrent streams per external location; guidance focuses on performance tuning (for example, using volumes for each subfolder), not on avoiding missed files.

So from a correctness perspective, having both a staging and production pipeline consume from the same external location and path is supported and will not cause one to steal events from the other.

Caveats to be aware of (These don’t change the “no shared cursor” answer, but are worth calling out:)

• Run frequency: If a given stream isn’t run for more than about 7 days, its stored read position in the cache can expire, forcing a full directory listing on the next run, but it should still not miss files because of that.
• Source cleanup options: Options like cloudFiles.cleanSource that delete source files after processing are not recommended when multiple streams read the same source, because the faster consumer can delete files before the slower one sees them. This is orthogonal to managed file events, but relevant for your staging vs production pattern.
• Normal streaming guarantees: As with any streaming job, keeping distinct checkpointLocation values per pipeline is critical. You’re already doing that, which is the right setup.

Direct answers to your questions

• Stored read position scope: It is per stream/pipeline, stored in that stream’s checkpoint. It is not a single shared cursor for all consumers of the external location.
• Multiple independent consumers: Yes. The managed file events cache is designed to support multiple independent streams (even across workspaces) reading the same file events without interfering with each other. The physical queue is shared per external location, but the logical read positions are per stream, much like Kafka consumer groups each having their own offsets.

Your current understanding - “each pipeline maintains its own read position via its checkpoint, making this safe even across workspaces” - is correct.

Hope this helps!

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.

Regards,