Databricks Community

jasperputs · ‎04-17-2023

I am trying to create a SQL Dashboard on top of a streaming dataset (Delta format). I created multiple queries referencing the file on the datalake, not a hive table. With these queries I created multiple visualizations in a Databricks SQL Dashboard. I added an automated refresh, as the new data in the streaming dataset needs to be displayed as fast as possible. This should refresh the data every 1 minute (More real time would be nice, but 1 minute works).

I would like to keep the dashboard open, so that I always have the visuals showing the data that has been refreshed less than one minute ago. However, the refresh keeps loading. The screenshot below shows that there is a refresh symbol going on at every visual, but the data is not being refreshed. In the screenshot it seems that the refresh is already taking more than 7 minutes.

If I click manually on the refresh button on the top right corner of the dashboard, the refresh takes never more than 10 seconds.

My question is, is this expected behaviour? Do we need to refresh the browser page before we can see the updates of the Dashboard Refresh? Am I doing something wrong? Or is this just a bug?

karthik_p · ‎04-17-2023

@Jasper Puts usually once you schedule your dashboard based on interval it should be bale to refresh

jasperputs · ‎04-17-2023

Well that is the problem. The refresh is scheduled, but if I keep the dashboard open it does not refresh automatically. I need to refresh my page before the dashboard refresh is displayed

Anonymous · ‎04-18-2023

@Jasper Puts :

Here are some potential solutions or next steps you can try:

Check the query refresh rate: Confirm that the queries are also set to refresh every minute or less. If the queries are not refreshing at the same rate as the dashboard, this could cause discrepancies between the data displayed in the queries and the data displayed in the dashboard.
Try a hard refresh of the dashboard: Pressing CTRL+F5 or SHIFT+F5 (depending on the browser) will do a hard refresh of the page, which may resolve any caching issues that are preventing the dashboard from updating automatically.
Check for errors or warnings: Check the Databricks logs for any errors or warnings related to the dashboard or queries. There may be an issue with the queries that is causing the dashboard to fail to update automatically.

jasperputs · ‎04-18-2023

@Suteja Kanuri Thanks for the response.

-There is no scheduled refresh on the queries. Is this something that is necessary? I assumed that refreshing the dashboard automatically will also refresh the query?

-I think the problem is linked to answer 2. It appears that the dataset keeps refreshing, but when I actually refresh the page, the dashboard is also refreshed. The goal is to display the dashboard somewhere on a tv/monitor. It would miss its purpose if a manual refresh needs to be done every time. Any ideas how this can be fixed? I tried it on both Chrome and Firefox.

Anonymous · ‎04-28-2023

@Jasper Puts :

Scheduled refresh on queries: It's not necessary to have a scheduled refresh on the queries if you are using automated refresh on the dashboard. The queries should be automatically refreshed when the dashboard refreshes.
Network connection: Yes, it's possible that the issue is related to the network connection. If the network connection is unstable or slow, it can cause delays in data transfer and refreshes. You can try optimizing your network connection by checking your network speed, increasing the bandwidth, or upgrading your internet plan.
Browser cache: Another possible issue could be related to your browser cache. Sometimes the browser cache can prevent the dashboard from refreshing properly. You can try clearing your browser cache or opening the dashboard in a private browsing window to see if that resolves the issue.
Dashboard settings: Finally, check the settings of the dashboard to ensure that the automatic refresh is enabled and set up correctly. You can also try adjusting the refresh frequency to see if that makes a difference. If none of these solutions work, it's possible that there may be a bug or technical issue with the dashboard