Ashwin_DSA
Databricks Employee
Databricks Employee

Thanks, @Malthe for checking that. 

Since I can't replicate your exact environment, I can only assume that the missing time and the associated DBU cost are probably due to parallelism. 

The fact that Photon accounts for 95% of the task time reinforces that it’s performing vectorised compute across a lot of data... and that the cluster is simply doing a large amount of work in parallel.

With regards to your earlier comment about the DBU cost, which is unaccounted for.... DBUs are a measure of compute used over time (roughly: cores × time × SKU factor). So if your query runs for a few minutes on a serverless cluster with multiple cores, it’s completely plausible to see something like 8 DBUs, even though:

  • the query only ran for a few minutes end‑to‑end, and
  • the UI shows 1.9 hours of aggregated task time.

In other words, those 8 DBUs are accounted for... they correspond to the parallel work the cluster did. It's just expressed in a different unit. If you want to sanity-check further, compare the bytes/rows processed in the query profile with the DBUs in the billing table for the same statement ID. If it still looks off, it’s worth opening a support ticket so the internal metrics for that specific run can be reviewed.

Hope this helps.

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.

Regards,
Ashwin | Delivery Solution Architect @ Databricks
Helping you build and scale the Data Intelligence Platform.
***Opinions are my own***