https://community.databricks.com/t5/data-engineering/how-to-clear-all-cache-without-restarting-the-cluster/m-p/24341#M16910 <PRE><CODE>%scala def clearAllCaching(tableName: Option[String] = None): Unit = { tableName.map { path =&gt; com.databricks.sql.transaction.tahoe.DeltaValidation.invalidateCache(spark, path) } spark.conf.set("com.databricks.sql.io.caching.bucketedRead.enabled", "false") spark.conf.set("spark.databricks.delta.smallTable.cache.enabled", "false") spark.conf.set("spark.databricks.delta.stats.localCache.maxNumFiles", "1") spark.conf.set("spark.databricks.delta.fastQueryPath.dataskipping.checkpointCache.enabled", "false") spark.conf.set("spark.databricks.io.cache.enabled", "false") com.databricks.sql.transaction.tahoe.DeltaLog.clearCache() spark.sql("CLEAR CACHE") sqlContext.clearCache() }</CODE></PRE><P>Please find the above piece of custom method to clear all the cache in the cluster without restarting . This will clear the cache by invoking the method given below.</P><PRE><CODE>%scala clearAllCaching()</CODE></PRE><P>The cache can be validated in the SPARK UI -&gt; storage tab in the cluster.</P> Tue, 29 Mar 2022 18:55:51 GMT shan_chandra 2022-03-29T18:55:51Z https://community.databricks.com/t5/data-engineering/how-to-clear-all-cache-without-restarting-the-cluster/m-p/24340#M16909 Tue, 29 Mar 2022 18:48:12 GMT https://community.databricks.com/t5/data-engineering/how-to-clear-all-cache-without-restarting-the-cluster/m-p/24340#M16909 shan_chandra 2022-03-29T18:48:12Z https://community.databricks.com/t5/data-engineering/how-to-clear-all-cache-without-restarting-the-cluster/m-p/24341#M16910 <PRE><CODE>%scala def clearAllCaching(tableName: Option[String] = None): Unit = { tableName.map { path =&gt; com.databricks.sql.transaction.tahoe.DeltaValidation.invalidateCache(spark, path) } spark.conf.set("com.databricks.sql.io.caching.bucketedRead.enabled", "false") spark.conf.set("spark.databricks.delta.smallTable.cache.enabled", "false") spark.conf.set("spark.databricks.delta.stats.localCache.maxNumFiles", "1") spark.conf.set("spark.databricks.delta.fastQueryPath.dataskipping.checkpointCache.enabled", "false") spark.conf.set("spark.databricks.io.cache.enabled", "false") com.databricks.sql.transaction.tahoe.DeltaLog.clearCache() spark.sql("CLEAR CACHE") sqlContext.clearCache() }</CODE></PRE><P>Please find the above piece of custom method to clear all the cache in the cluster without restarting . This will clear the cache by invoking the method given below.</P><PRE><CODE>%scala clearAllCaching()</CODE></PRE><P>The cache can be validated in the SPARK UI -&gt; storage tab in the cluster.</P> Tue, 29 Mar 2022 18:55:51 GMT https://community.databricks.com/t5/data-engineering/how-to-clear-all-cache-without-restarting-the-cluster/m-p/24341#M16910 shan_chandra 2022-03-29T18:55:51Z https://community.databricks.com/t5/data-engineering/how-to-clear-all-cache-without-restarting-the-cluster/m-p/24342#M16911 <P>Great. You can add ​ spark.catalog.clearCache() as well.</P> Tue, 29 Mar 2022 19:53:26 GMT https://community.databricks.com/t5/data-engineering/how-to-clear-all-cache-without-restarting-the-cluster/m-p/24342#M16911 Hubert-Dudek 2022-03-29T19:53:26Z