- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-28-2022 05:54 AM
Does Spark SQL have both materialized and non-materialized views? With materialized views, it reads from cache for unchanged data, and only from the table for new/changed rows since the view was last accessed? Thanks!
- Labels:
-
Spark sql
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-28-2022 07:13 AM
Delta cache is per file so if you have dataset which data structure is splited in 100 files and 1 file was updated theoretically that 1 only should be evicted. It is automatic.
There is also Spark cache which is totally manual so you control persist/cache operation.
When you use Databricks SQL endpoint delta cache is automatically handled, for delta cache optimized VMS is enabled by default for other by setting it in spark config.
My blog: https://databrickster.medium.com/
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-31-2022 03:42 PM
@Mark Hummel - Does Hubert's answer resolve your question?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-01-2022 05:22 AM
Yes
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-01-2022 07:57 AM
AWESOME!
