I am wondering if it is using Remote result cache, in that case the config should work.
There are 4 types of cache mentioned here https://docs.databricks.com/en/sql/user/queries/query-caching.html#types-of-query-caches-in-databric...
Local cache: once the cluster is stopped or restarted, the cache is cleaned and all query results are removed.
Databricks SQL UI cache: the cache is invalidated once the underlying tables have been updated.
You can delete query results by re-running the query that you no longer want to be stored. Once re-run, the old query results are removed from cache.
Disk cache: stores data on disk, allowing for accelerated data reads.
disk cache automatically detects changes to the underlying data files. When it detects changes, the cache is invalidated. The disk cache shares the same lifecycle characteristics as the local result cache. This means that when the cluster is stopped or restarted, the cache is cleaned and needs to be repopulated.
Even when I stop my serverless warehouse cluster and start it again it uses cache.
So, that only leaves us to Remote result cache : persisting them as workspace system data. This cache is not invalidated by the stopping or restarting of a SQL warehouse.
SET use_cached_result = false;
This command disables the query result caching for the current session. By any chance you are using it in a different session? If yes, before you run your command, set this.