cancel
Showing results for 
Search instead for 
Did you mean: 
Warehousing & Analytics
cancel
Showing results for 
Search instead for 
Did you mean: 

Do SQL Endpoints cache query results?

User16826992666
Valued Contributor
 
1 ACCEPTED SOLUTION

Accepted Solutions

User16826992666
Valued Contributor

SQL Analytics actually uses several layers of caching. Some documentation about the different layers can be found here in the documentation. 

There are two primary layers that users will experience. 

1) The first is that the actual data results of specific queries are stored in memory for subsequent query runs. So if you run the same query twice, it won't have to recompute at all.

2) The second type is delta caching, which is where actual copies of the files that are read from data storage are created on the local SSD's on the endpoint clusters. Then when those files are needed in subsequent queries the cached files on the SSD's are accessed rather than going back and pulling the data from the source again. With this level of caching, you are actually making a copy of the data you are querying on the SSD's of the cluster.

View solution in original post

1 REPLY 1

User16826992666
Valued Contributor

SQL Analytics actually uses several layers of caching. Some documentation about the different layers can be found here in the documentation. 

There are two primary layers that users will experience. 

1) The first is that the actual data results of specific queries are stored in memory for subsequent query runs. So if you run the same query twice, it won't have to recompute at all.

2) The second type is delta caching, which is where actual copies of the files that are read from data storage are created on the local SSD's on the endpoint clusters. Then when those files are needed in subsequent queries the cached files on the SSD's are accessed rather than going back and pulling the data from the source again. With this level of caching, you are actually making a copy of the data you are querying on the SSD's of the cluster.

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.