Table update trigger and File Arrival trigger latency

[MikeGo]

Hi team,

When using table update or file arrival trigger, what latency I can expect for the trigger. Does Databricks poll the source by some schedule? If yes, whether the poll is free?

Thanks

[Sumit_7]

@MikeGo  File arrival triggers work near-real time and rely on polling unless you use notifications. No charge for polling, but cloud storage API calls are billed. Check this thread - Solved: File Arrival Trigger - Reduce Listing - Databricks Community - 119987
Try searching your query first, mostly there's a chance that someone had already asked it before.

[Ashwin_DSA]

Hi @MikeGo,

For both file arrival triggers and table update triggers, Databricks makes a best-effort check roughly every minute to detect new files or table updates. The end-to-end latency you should plan for is therefore on the order of 1-2 minutes, plus any time needed to start the job/cluster. There is no separate Databricks charge for these triggers themselves. The only incremental cost is from the cloud provider APIs used under the hood (for example, listing storage or reading table metadata). If you need sub-minute SLAs, we typically recommend a continuous stream (e.g., an Auto Loader with a short processing trigger) rather than relying solely on triggers.

Here is an official reference if you'd like to read further. 

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.

[MikeGo]

Hi @Ashwin_DSA ,

Thanks for reply. Our scenario is we have a Kafka source streaming data to raw table, and then downstream tasks consume the raw data. The streaming to raw is sparse at minutes level. If we create streaming job for downstream, it might use thousands of dollars a year for EC2 even if we use single node cluster, plus the DBU. As source is sparse, a streaming job is a waste here.

We want to run downstream tasks at minutes level. Obviously we cannot use all purpose cluster because of cold start. So we think we can use serverless + table update trigger. Is below understanding correct:

1. When using table update triggered by raw table, if data keeps coming, there is actually no big benefit for table update trigger+serverless vs. a minutes level cron+serverless right?

2. For table update trigger or file arrival trigger without file events, actually behind the scene, Databricks is doing poll. For table update trigger or file arrival trigger with file events, actually it is push model? 

3. We didn't find specific latency number for trigger with vs. without file events. What latency num we can tell our stakeholders based on triggers?

4. The poll part or getting notification from file events are free. The resource usage is our cost.

5. We stream Kafka data to a raw table, and downstream monitor the raw table. We noticed the first time trigger needs a long initialization if raw has many commit versions (as it is sink from Kafka). However sometimes we get `The table 'xxx' has exceeded the maximum number of initial evaluations (20). This can occur if the table's delta log directory is large and contains many versions. Consider running VACUUM on the table to clean up old versions.` So looks like the trigger initialization checks early commit versions. If yes, why? Shouldn't be the new pipeline just runs immediately without checking early logs?

[Ashwin_DSA]

Hi @MikeGo,

Please see the responses below for each question.

1. When using table update triggered by raw table, if data keeps coming, there is actually no big benefit for table update trigger+serverless vs. a minutes level cron+serverless right?

You're right that the cost advantage largely disappears when the source is continuously producing commits. A table update trigger only saves money when it can skip runs, so under steady traffic, both patterns will fire at roughly the same cadence and consume roughly the same compute. That said, the trigger still earns its keep in two subtle ways even under constant flow. First, it gives you the "Wait after last change" knob, which resets on each new commit and lets you debounce a burst of writes into a single downstream run, something a cron cannot do. Second, the trigger hands your job a parameter payload with the updated table list and the most recent commit version, so your downstream logic can be cleanly incremental without you tracking watermarks yourself. If neither of those matters for your workload, a cron is simpler and costs about the same.

2. For table update trigger or file arrival trigger without file events, actually behind the scene, Databricks is doing poll. For table update trigger or file arrival trigger with file events, actually it is push model? 

Your instinct is mostly right, with one nuance worth calling out. Table update triggers and file arrival triggers without file events are genuinely poll based, meaning the Databricks control plane periodically reads the Delta log or lists the storage directory to detect change. When you enable file events, it's actually a hybrid rather than pure push. The cloud provider pushes change notifications (S3 to SNS/SQS, ADLS to Event Grid) into a Databricks-managed file events service that caches the metadata, and then your trigger still periodically queries that cache rather than the raw directory. So the heavy listing work is eliminated, but the trigger itself continues to run on an evaluation loop on the Databricks side.

3. We didn't find specific latency number for trigger with vs. without file events. What latency num we can tell our stakeholders based on triggers?

The docs phrase it as "best effort to check for new files every minute," and the table update trigger uses the same underlying evaluation engine, so the honest number to give stakeholders is roughly one to two minutes from data change to job start, plus serverless cold start on top of that. Serverless jobs cold start is typically in the 15 to 25 second range, so realistically you're looking at about one to two and a half minutes end to end. File events do not give you an order of magnitude improvement in latency. What they buy you is scale and robustness, specifically removing the 10K file directory cap and the 50 jobs per workspace cap, and avoiding expensive listings on large directories. There is no formal external SLO on trigger latency today, so the fair way to describe it is near real time at minute granularity, not hard real time in seconds.

4. The poll part or getting notification from file events are free. The resource usage is our cost.

Yes, correct. Databricks does not charge DBUs for the trigger itself. The docs state that file arrival and table update triggers do not incur additional costs beyond the cloud provider's charges for listing and reading metadata. You will pay cloud costs for object listings or notification infrastructure (SNS, SQS, Event Grid), which are negligible at typical volumes, and you pay DBUs only once a job actually runs. This is exactly why the serverless plus trigger pattern is so cost effective for sparse sources.

5. We stream Kafka data to a raw table, and downstream monitor the raw table. We noticed the first time trigger needs a long initialization if raw has many commit versions (as it is sink from Kafka). However sometimes we get `The table 'xxx' has exceeded the maximum number of initial evaluations (20). This can occur if the table's delta log directory is large and contains many versions. Consider running VACUUM on the table to clean up old versions.` So looks like the trigger initialization checks early commit versions. If yes, why? Shouldn't be the new pipeline just runs immediately without checking early logs?

When you set up a table update trigger, the first run has no prior state for that table, so it needs to walk the Delta log in chunks to establish a consistent baseline and make sure it does not miss relevant commits. To protect the control plane, Databricks caps that initialisation at 20 evaluations. On a Kafka sink, which typically produces many small, frequent commits, the log can easily have thousands of versions between checkpoints, and the baseline walk can exceed the cap and throw the error you saw. The error message suggests VACUUM, but that is slightly misleading on its own. VACUUM only removes data files and does not shrink the _delta_log directory. The actual log cleanup is governed by checkpoints (written every 10 commits by default) and delta.logRetentionDuration (30 days by default). The practical fixes are to make sure checkpoints are being written and to keep log retention reasonable, or for a faster escape hatch, introduce a thin staging table between the Kafka sink and your downstream, point the trigger at the staging table, and let it start with a clean log. Increasing the micro batch interval on the Kafka to raw job also helps because it produces fewer, larger commits and reduces log churn at the source.

To give you a simple answer... for a sparse Kafka source feeding a raw Delta table with minute-level downstream needs, serverless compute plus a table update trigger is a good fit. Set a minimum time between triggers in the 60 to 300 second range, optionally add a short "wait after last change" to coalesce bursts, keep the raw table's Delta log healthy so initialisation stays fast, and accept a one to two minute plus cold start latency when describing the SLA to stakeholders. If the raw table's log depth becomes hard to control, a staging table as the trigger source is the cleanest workaround.

Hope this helps.

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.

[MikeGo]

Hi @Ashwin_DSA ,

Thanks for the input. This is very helpful. For the last question, we thought about to create another table as staging, which is specifically used as trigger. Any time source has changes, we will update the staging table too. However to be strict we need to make sure the loading to source and staging are in one transaction and this is hard. Finally we gave up this. Now seems with

```

ALTER TABLE xxx SET TBLPROPERTIES (
'delta.deletedFileRetentionDuration' = 'interval 1 hours',
'delta.logRetentionDuration' = 'interval 1 hours'
)

```

The log count is reduced. The table update trigger can be initialized. And we can reset the properties back after that. 

[Ashwin_DSA]

Hi @MikeGo,

The two properties together do exactly what you want, but they only set the eligibility threshold for cleanup. They don't actively delete anything themselves. Old log files get pruned when the next checkpoint is written (every 10 commits by default), and old data files only go away when you actually run VACUUM. So if you set the properties and your raw table is still receiving commits from Kafka, the next checkpoint will trigger the log pruning, and your trigger
initialisation should succeed shortly after. If the table is idle, you may need to force a small commit or run VACUUM to trigger the cleanup.

The bigger thing to be careful about is that delta.deletedFileRetentionDuration = 1 hour is genuinely aggressive. Anything that reads the table with a state pointing to a version older than one hour will break. Concretely, that includes any structured streaming consumer of the raw table whose checkpoint is lagging by more than an hour, any time travel queries (VERSION AS OF, TIMESTAMP AS OF), any CDF reads against older versions, and any other downstream trigger that was relying on commit history. If your downstream is fully serverless and lightweight, this is probably fine, but in a busy environment with multiple consumers, it can cause cascading failures. Resetting the properties back afterwards is the right move, and I would add running a VACUUM once before resetting, so the cleanup actually completes within the short retention window.

For ongoing hygiene rather than a one-shot fix, the more sustainable approach is to leave delta.logRetentionDuration at something modest like one to seven days and ensure checkpoints are being written regularly. The Kafka to raw streaming job's micro-batch interval is the lever there. Larger, less frequent batches reduce log churn at the source and keep the trigger initialisation fast without you having to flip table properties on and off. See Delta table properties and Work with Delta Lake table history for the full set of related settings.

On the staging table idea you abandoned, you were right to be uncomfortable about the two-write transaction problem. Worth noting that you can sidestep it entirely by reading the raw table's commits with Delta change data feed into the staging table as a streaming job, so the staging table is derived from raw rather than written in parallel with it. Atomicity stops mattering because there's only one source of truth. That said, if the property toggle is working,
sticking with it is simpler.

Hope this helps.

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.

[MikeGo]

Hi @Ashwin_DSA ,

Appreciate for the further clarification. Let's make this even clearer. 

"the trigger hands your job a parameter payload with the updated table list and the most recent commit version"

This is a good thing but likely it cannot be used, especially the downstream has no idempotent sink and expect exactly-once delta. Imagine a job gets the current version, read the delta and failed to sink. So in practice maybe we cannot use it.

The initialization is slow hints it tries to go through all commit versions. And table update trigger returns commit version might be one of the reasons. If returning version or timestamp is not that useful, maybe table update trigger should not be designed as return that and go through all versions. At least user should be given an option to skip it. If user has a table as streaming sink, very likely it has many versions and new pipeline with table update trigger cannot get initialized. If I just deploy a new pipeline and monitor a table update, it doesn't make sense to go through all versions to find whether it should get triggered. The new pipeline should be triggered immediately or only check wether there is change after it is created. 

We are aware of delta.deletedFileRetentionDuration = 1 hour may break the state. However we don't do time travel stuff. Everything we need is to read the table directly, and we will reset this one back after initialization is done.

"Worth noting that you can sidestep it entirely by reading the raw table's commits with Delta change data feed into the staging table as a streaming job, so the staging table is derived from raw rather than written in parallel with it. "

I think Delta Table has some design issues - it is super slow to get changes between 2 versions. It has checkpoint, so it is fast to get version snapshot at any timing point. However, it is super slow to get changes between 2 versions. Say I have a raw table to load from Kafka, and then I want to read it by batch. Either the batch is using CDF or by SS (with availableNow), getting the changes is super slow. The slowness has nothing related with data volume. For 15000 versions with just 15000 rows, it is still super slow, or in SS, even if it readStream and filter with where 1=2, it still needs 20~40 min to go through each version. For Delta Table, anything related with many versions is slow. The initialization failure is similar. SS is even worse than CDF because looks like it iterates the changes twice and do schema validation etc. and totally kills the availableNow performance. 

Because of this we did Kafka -> Raw -> Staging -> Downstream. From Raw to Staging is a streaming job and we can get commit_version. Then we can create LC on commit_version and getting the delta changes easily. Downstream monitors Staging. However as staging is loaded by SS so it still has many versions. The initialization failure is actually from here. That is also the reason we don't need time travel for staging. 

BTW, iceberg should not have this perf issues to read multi versions.

Thanks