Databricks SQL Warehouse did not auto stop after specified 90 minute interval - why not?

[rendorHaevyn]

In this specific case, we're running a 2XSmall SQL Warehouse on Databricks SQL.

In looking at the SQL Warehouse monitoring log for this cluster, we noticed:

• final query executed by user at 10:26 on 2023-06-20
• no activity for some time, yet cluster remained active
• subsequent user query executed at 21:33 on 2023-06-20

Now, between these queries, the SQL Warehouse should have shut down due to inactivty after 90 minutes.

We have not observed this occurring over the preceding 12 months.

A key variable which has changed is that Databricks recently allowed users to connect to SQL Warehouses in the Data Engineering (Notebooks) service, and this may have occurred, and this may have kept this cluster active, even though no queries were instantiated against the cluster.

Databricks - any feedback on this?

[Anonymous]

Hi @Cameron McPherson​ 

Great to meet you, and thanks for your question!

Let's see if your peers in the community have an answer to your question. Thanks.

[Michael42]

I recently had this problem on Azure. Created a SQL Warehouse with a 20 minute auto stop setting. When I saw a $1500 bill on my cc in December, I checked on Azure and saw that the SQL Warehouse was up and running since October 8th. I tried to shut it down manually and it just kept running again. I doubled checked the auto stop configuration setting and it was still twenty minutes.

I looked under the query history and I saw that every few seconds  a USE command was failing trying to connect to a database that had been deleted some time ago. I think that somehow this error was preventing the inactivity clock from ticking down. I had to actually delete the SQL Warehouse to make it stop.

I only use the Databricks instance for study and had barely been on it in months.

The damage is an $2800 bill that I am still struggling with Microsoft support on because they don't seem to understand how Databricks costs are supposed to work. If or when I resolve this with them I'll get back on here. I guess I could try to reproduce it, but it should be obvious to them that Databricks and VM costs shot up together, and back down together as soon as I deleted the SQL Warehouse. They seem to think I allocated a VM service, and my VM costs are from that. Will try with them, again tomorrow. 

[Emil_Kaminski]

@Michael42 sounds like some sort of horror story. Let us know how it goes. 

It happens to me as well, but I was lucky enough to have this situation on very small compute cluster for just couple of days.