AI/BI Dashboard refresh via DABs + Jobs executes successfully but dashboard does not update without

[mnissen1337]

I’m migrating a solution from an on-prem setup to Databricks AI/BI Dashboards, and I’m trying to replicate a near real-time dashboard experience (around ~1 minute latency is acceptable).

In the legacy setup, we used DirectQuery combined with automatic page refresh, which allowed visuals to update continuously without manual intervention.

What I’ve tried

1. Dashboard refresh via job (DAB task)
   • The job executes successfully every minute
   • However, the dashboard visuals do not reflect updated data afterwards
2. Scheduled refresh on published dashboard
   • The schedule runs as expected
   • But the dashboard still does not update in the UI unless I manually refresh the browser (F5)

Questions

Is this expected behavior for AI/BI dashboards in Databricks? That is, is it really not possible to have a "live" view of a dashboard without having to manually refersh the page even when a schedule is configured?

[mnissen1337]

For some reason, I could not attach the DAB resource definitions. Here they are: 

[amirabedhiafi · Accepted solution]

Hi @mnissen1337  !

Yes I think this behavior is expected because a dashboard schedule or dashboard_task refresh runs the dashboard dataset SQL and refreshes the query result cache. It is mainly for keeping cached results warm, improving load time and producing subscription snapshots. It doesn't behave like PBO DQ with automatic page refresh and it does not push updated results into an open browser session. Because this feature does the running of the dataset SQL and populate the query result cache.

So your DAB job can succeed every minute but users who already have the dashboard open may still see the old data until they click refresh or reload the browser.

As a workaround try to embed the dashboard in a DBKS app and reload the page every 60 sec.

[mnissen1337]

Thanks for the answer! Thats unfortunate. Do you think in the future Databricks will support the provided use case or will we need to do workarounds such as embedding the Dashboard in an DBKS app or maybe just create the entire Dashboard in an app using e.g. Plotly or something similiar 🙂

[amirabedhiafi · Accepted solution]

Hi again !

Honestly I wouldn’t rely on this becoming available unless DBKS confirms as part of the public roadmap.

And as I mentioned above, for business reporting I would keep AI/BI Dashboards and use scheduled refresh to keep the cache warm and for near real time operational monitoring, I would either embed the AI/BI Dashboard in a DBKS app and reload the iframe every 60 seconds or build the dashboard directly in a DBKS app using plotly (you can use streamlit, dash..)

or simply use Power BI 🙂