Hi @MikeGo,

Please see the responses below for each question.

1. When using table update triggered by raw table, if data keeps coming, there is actually no big benefit for table update trigger+serverless vs. a minutes level cron+serverless right?

You're right that the cost advantage largely disappears when the source is continuously producing commits. A table update trigger only saves money when it can skip runs, so under steady traffic, both patterns will fire at roughly the same cadence and consume roughly the same compute. That said, the trigger still earns its keep in two subtle ways even under constant flow. First, it gives you the "Wait after last change" knob, which resets on each new commit and lets you debounce a burst of writes into a single downstream run, something a cron cannot do. Second, the trigger hands your job a parameter payload with the updated table list and the most recent commit version, so your downstream logic can be cleanly incremental without you tracking watermarks yourself. If neither of those matters for your workload, a cron is simpler and costs about the same.

2. For table update trigger or file arrival trigger without file events, actually behind the scene, Databricks is doing poll. For table update trigger or file arrival trigger with file events, actually it is push model? 

Your instinct is mostly right, with one nuance worth calling out. Table update triggers and file arrival triggers without file events are genuinely poll based, meaning the Databricks control plane periodically reads the Delta log or lists the storage directory to detect change. When you enable file events, it's actually a hybrid rather than pure push. The cloud provider pushes change notifications (S3 to SNS/SQS, ADLS to Event Grid) into a Databricks-managed file events service that caches the metadata, and then your trigger still periodically queries that cache rather than the raw directory. So the heavy listing work is eliminated, but the trigger itself continues to run on an evaluation loop on the Databricks side.

3. We didn't find specific latency number for trigger with vs. without file events. What latency num we can tell our stakeholders based on triggers?

The docs phrase it as "best effort to check for new files every minute," and the table update trigger uses the same underlying evaluation engine, so the honest number to give stakeholders is roughly one to two minutes from data change to job start, plus serverless cold start on top of that. Serverless jobs cold start is typically in the 15 to 25 second range, so realistically you're looking at about one to two and a half minutes end to end. File events do not give you an order of magnitude improvement in latency. What they buy you is scale and robustness, specifically removing the 10K file directory cap and the 50 jobs per workspace cap, and avoiding expensive listings on large directories. There is no formal external SLO on trigger latency today, so the fair way to describe it is near real time at minute granularity, not hard real time in seconds.

4. The poll part or getting notification from file events are free. The resource usage is our cost.

Yes, correct. Databricks does not charge DBUs for the trigger itself. The docs state that file arrival and table update triggers do not incur additional costs beyond the cloud provider's charges for listing and reading metadata. You will pay cloud costs for object listings or notification infrastructure (SNS, SQS, Event Grid), which are negligible at typical volumes, and you pay DBUs only once a job actually runs. This is exactly why the serverless plus trigger pattern is so cost effective for sparse sources.

5. We stream Kafka data to a raw table, and downstream monitor the raw table. We noticed the first time trigger needs a long initialization if raw has many commit versions (as it is sink from Kafka). However sometimes we get `The table 'xxx' has exceeded the maximum number of initial evaluations (20). This can occur if the table's delta log directory is large and contains many versions. Consider running VACUUM on the table to clean up old versions.` So looks like the trigger initialization checks early commit versions. If yes, why? Shouldn't be the new pipeline just runs immediately without checking early logs?

When you set up a table update trigger, the first run has no prior state for that table, so it needs to walk the Delta log in chunks to establish a consistent baseline and make sure it does not miss relevant commits. To protect the control plane, Databricks caps that initialisation at 20 evaluations. On a Kafka sink, which typically produces many small, frequent commits, the log can easily have thousands of versions between checkpoints, and the baseline walk can exceed the cap and throw the error you saw. The error message suggests VACUUM, but that is slightly misleading on its own. VACUUM only removes data files and does not shrink the _delta_log directory. The actual log cleanup is governed by checkpoints (written every 10 commits by default) and delta.logRetentionDuration (30 days by default). The practical fixes are to make sure checkpoints are being written and to keep log retention reasonable, or for a faster escape hatch, introduce a thin staging table between the Kafka sink and your downstream, point the trigger at the staging table, and let it start with a clean log. Increasing the micro batch interval on the Kafka to raw job also helps because it produces fewer, larger commits and reduces log churn at the source.

To give you a simple answer... for a sparse Kafka source feeding a raw Delta table with minute-level downstream needs, serverless compute plus a table update trigger is a good fit. Set a minimum time between triggers in the 60 to 300 second range, optionally add a short "wait after last change" to coalesce bursts, keep the raw table's Delta log healthy so initialisation stays fast, and accept a one to two minute plus cold start latency when describing the SLA to stakeholders. If the raw table's log depth becomes hard to control, a staging table as the trigger source is the cleanest workaround.

Hope this helps.

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.