topic Re: Simple Question in Data Engineering

Simple Question

MRH — Fri, 28 Jan 2022 13:54:56 GMT

Does Spark SQL have both materialized and non-materialized views? With materialized views, it reads from cache for unchanged data, and only from the table for new/changed rows since the view was last accessed? Thanks!

Re: Simple Question

Hubert-Dudek — Fri, 28 Jan 2022 15:13:36 GMT

Delta cache is per file so if you have dataset which data structure is splited in 100 files and 1 file was updated theoretically that 1 only should be evicted. It is automatic.

There is also Spark cache which is totally manual so you control persist/cache operation.

When you use Databricks SQL endpoint delta cache is automatically handled, for delta cache optimized VMS is enabled by default for other by setting it in spark config.

Re: Simple Question

Anonymous — Mon, 31 Jan 2022 23:42:00 GMT

@Mark Hummel - Does Hubert's answer resolve your question?

Re: Simple Question

MRH — Tue, 01 Feb 2022 13:22:19 GMT

Yes

Re: Simple Question

Anonymous — Tue, 01 Feb 2022 15:57:53 GMT

AWESOME!