Databricks Cache Options

[sk_databricks]

Hi,

We are working on Databricks solution hosted on AWS. We are exploring the caching options in Databricks. Apart from the Databricks cache and spark cache? What are the options? 

Is it feasible to use 3rd party Cache solutions like AWS Elastic Cache for Redis?

[Walter_C]

Databricks provides several caching options to enhance performance by minimizing Input and Output (I/O) read and write operations. These include:

1. Databricks Disk Cache: This cache accelerates data reads by creating copies of remote Parquet data files in nodes’ local storage using a fast intermediate data format. The data is cached automatically whenever a file has to be fetched from a remote location. Successive reads of the same data are then performed locally, which results in significantly improved reading speed. This cache is recommended over Spark caching as it provides better performance outcomes.

2. Spark Cache: Spark provides an optimization mechanism to cache the intermediate computation of a Spark DataFrame so they can be reused in subsequent actions. You can also cache a table using the CACHE TABLE command. There are different cache modes that allow you to choose where to store the cached data (in the memory, in the disk, in the memory and the disk, with or without serialization, etc.).

3. Query Caching in Databricks SQL: This caching can significantly speed up query execution and minimize warehouse usage, resulting in lower costs and more efficient resource utilization. It includes User Interface Cache, Result Cache (Local and Remote), and Disk Cache.