What is Databricks Delta Cache? and how does it help improving performance over Apache Spark cache

Srikanth_Gupta_ — Mon, 14 Jun 2021 20:24:44 GMT

Re: What is Databricks Delta Cache? and how does it help improving performance over Apache Spark cache

sajith_appukutt — Thu, 17 Jun 2021 22:05:58 GMT

Delta cache is an automatic hands-free solution that leverages high read speeds of modern SSDs to transparently create copies of remote files in nodes’ local storage to accelerate data reads . In comparison, you have choose what and when to cache with Spark cache.

Also

Delta cache can be read and operated on faster than the data in the Spark cache.
Eviction happens automatically on file changes

More details could be found at https://docs.databricks.com/delta/optimizations/delta-cache.html

topic Re: What is Databricks Delta Cache? and how does it help improving performance over Apache Spark cache in Data Engineering

What is Databricks Delta Cache? and how does it help improving performance over Apache Spark cache

Re: What is Databricks Delta Cache? and how does it help improving performance over Apache Spark cache