cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Databricks Cache Options

sk_databricks
New Contributor

Hi,

We are working on Databricks solution hosted on AWS. We are exploring the caching options in Databricks. Apart from the Databricks cache and spark cache? What are the options? 

Is it feasible to use 3rd party Cache solutions like AWS Elastic Cache for Redis?

1 REPLY 1

Walter_C
Databricks Employee
Databricks Employee

Databricks provides several caching options to enhance performance by minimizing Input and Output (I/O) read and write operations. These include:

  1. Databricks Disk Cache: This cache accelerates data reads by creating copies of remote Parquet data files in nodesโ€™ local storage using a fast intermediate data format. The data is cached automatically whenever a file has to be fetched from a remote location. Successive reads of the same data are then performed locally, which results in significantly improved reading speed. This cache is recommended over Spark caching as it provides better performance outcomes.

  2. Spark Cache: Spark provides an optimization mechanism to cache the intermediate computation of a Spark DataFrame so they can be reused in subsequent actions. You can also cache a table using the CACHE TABLE command. There are different cache modes that allow you to choose where to store the cached data (in the memory, in the disk, in the memory and the disk, with or without serialization, etc.).

  3. Query Caching in Databricks SQL: This caching can significantly speed up query execution and minimize warehouse usage, resulting in lower costs and more efficient resource utilization. It includes User Interface Cache, Result Cache (Local and Remote), and Disk Cache.

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local communityโ€”sign up today to get started!

Sign Up Now