Simple Question

[MRH]

Does Spark SQL have both materialized and non-materialized views? With materialized views, it reads from cache for unchanged data, and only from the table for new/changed rows since the view was last accessed? Thanks!

[Hubert-Dudek · Accepted solution]

Delta cache is per file so if you have dataset which data structure is splited in 100 files and 1 file was updated theoretically that 1 only should be evicted. It is automatic.

There is also Spark cache which is totally manual so you control persist/cache operation.

When you use Databricks SQL endpoint delta cache is automatically handled, for delta cache optimized VMS is enabled by default for other by setting it in spark config.

[Anonymous]

@Mark Hummel​ - Does Hubert's answer resolve your question?

[MRH]

Yes

[Anonymous]

AWESOME!