09-11-2024 01:16 AM - edited 09-11-2024 01:28 AM
Hello,
Following abnormally high costs when using serverless sql on September 9 and 10, I noticed that the cluster sometimes stays on for an hour even though it's not receiving any new requests, and that the auto-stop is set to 5 minutes of inactivity.
Is there an explanation for this?
Thank you for your help.
09-11-2024 01:26 AM
09-11-2024 01:30 AM
Hi @EmmaP
Have you checked the monitoring tab?
Any active sessions, queued queries that could block cluster from terminating?
Monitor a SQL warehouse - Azure Databricks | Microsoft Learn
09-11-2024 02:37 AM
Yes I checked the monitoring tab. I don't see any active or queued query after 12:05. And none of the previous queries (executed before 12:05) took longer than 10 seconds.
09-16-2024 06:01 AM
I have the impression that the cluster's auto-stop (set to turn off the cluster after 5 minutes of inactivity) does not work when the last request was sent from Power BI Services. Has anyone encountered this issue?
09-16-2024 09:19 AM - edited 09-16-2024 09:30 AM
Hi @EmmaP!
I have encountered this. Even though the UI says that they are complete, they actually are not. While the query itself completed, the client is still fetching the data from the SQL Warehouse.
To check if this is your issue, from the monitoring page (shown in your screenshot) click one of the queries to open the details panel. Then check the "Result fetching by client" metric. While queries are in progress (or show as "Finished") -- If it's null or doesn't show a value, that means the client is still fetching the result.
This seems to really be a problem with Power BI - for some reason it leaves the connection open. I'm not sure if it actually completes fetching the data. Databricks kills these connections after one hour -- which is what I think you're seeing in your screenshot. After the one-hour timeout, the query should swap (I think) from "Finished" to "Failed" and show the error "Query has been timed out due to inactivity."
To combat this, I have a workflow that runs every hour to check if any queries are stuck fetching data -- and if so, kill the queries.
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group