cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Simple Question

MRH
New Contributor II

Does Spark SQL have both materialized and non-materialized views? With materialized views, it reads from cache for unchanged data, and only from the table for new/changed rows since the view was last accessed? Thanks!

1 ACCEPTED SOLUTION

Accepted Solutions

Hubert-Dudek
Esteemed Contributor III

Delta cache is per file so if you have dataset which data structure is splited in 100 files and 1 file was updated theoretically that 1 only should be evicted. It is automatic.

There is also Spark cache which is totally manual so you control persist/cache operation.

When you use Databricks SQL endpoint delta cache is automatically handled, for delta cache optimized VMS is enabled by default for other by setting it in spark config.

View solution in original post

4 REPLIES 4

Hubert-Dudek
Esteemed Contributor III

Delta cache is per file so if you have dataset which data structure is splited in 100 files and 1 file was updated theoretically that 1 only should be evicted. It is automatic.

There is also Spark cache which is totally manual so you control persist/cache operation.

When you use Databricks SQL endpoint delta cache is automatically handled, for delta cache optimized VMS is enabled by default for other by setting it in spark config.

Anonymous
Not applicable

@Mark Hummel​ - Does Hubert's answer resolve your question?

MRH
New Contributor II

Yes

Anonymous
Not applicable

AWESOME!

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.