topic Re: Disabling query caching for SQL Warehouse in Data Engineering

Disabling query caching for SQL Warehouse

Marvin_T — Wed, 07 Jun 2023 07:39:34 GMT

Hello everybody,

I am currently trying to run some performance tests on queries in Databricks on Azure. For my tests, I am using a Classic SQL Warehouse in the SQL Editor.

I have created two views that contain the same data but have different structure -> I am trying to see if those different structures affect query performance.

My issue:

To test the performance I am running similiar queries that result in the same output for the individual views. Caching of course screws with the results as the output is fully cached from the previous query.

I have tried to do the following things already:

SET use_cached_result = false; -> Does not impact my query caching unfortunately
CLEAR CACHE; -> Does not work on SQL Warehouses
spark.conf.set("spark.databricks.io.cache.enabled", "false") -> Also doesnt impact the query caching

Looking at the caching structure of databricks I guess I need to disable the UI Cache, but I can't find any information on how to do so. Does anybody know how to solve my problem?

Re: Disabling query caching for SQL Warehouse

-werners- — Wed, 07 Jun 2023 08:16:45 GMT

besides caching, are you sure the queries execute a different query plan?

I'd think (not sure though) that if you stop the warehouse, the cache gets cleared. Perhaps that is useable?

Re: Disabling query caching for SQL Warehouse

Marvin_T — Wed, 07 Jun 2023 08:38:14 GMT

They are probably executing the same query plan now that you say it. And yes, restarting the warehouse does theoretically works but it isnt a nice solution.

I guess I will do some restarting and build averages to have a good comparison for now

Re: Disabling query caching for SQL Warehouse

TessinPeter — Wed, 25 Jun 2025 11:09:31 GMT

Just restarting the warehouse (or using a different one) does not solve the problem for me.

@Marvin_T does it work for you?