โ03-29-2022 11:48 AM
โ03-29-2022 11:55 AM
%scala
def clearAllCaching(tableName: Option[String] = None): Unit = {
tableName.map { path =>
com.databricks.sql.transaction.tahoe.DeltaValidation.invalidateCache(spark, path)
}
spark.conf.set("com.databricks.sql.io.caching.bucketedRead.enabled", "false")
spark.conf.set("spark.databricks.delta.smallTable.cache.enabled", "false")
spark.conf.set("spark.databricks.delta.stats.localCache.maxNumFiles", "1")
spark.conf.set("spark.databricks.delta.fastQueryPath.dataskipping.checkpointCache.enabled", "false")
spark.conf.set("spark.databricks.io.cache.enabled", "false")
com.databricks.sql.transaction.tahoe.DeltaLog.clearCache()
spark.sql("CLEAR CACHE")
sqlContext.clearCache()
}
Please find the above piece of custom method to clear all the cache in the cluster without restarting . This will clear the cache by invoking the method given below.
%scala clearAllCaching()
The cache can be validated in the SPARK UI -> storage tab in the cluster.
โ03-29-2022 11:55 AM
%scala
def clearAllCaching(tableName: Option[String] = None): Unit = {
tableName.map { path =>
com.databricks.sql.transaction.tahoe.DeltaValidation.invalidateCache(spark, path)
}
spark.conf.set("com.databricks.sql.io.caching.bucketedRead.enabled", "false")
spark.conf.set("spark.databricks.delta.smallTable.cache.enabled", "false")
spark.conf.set("spark.databricks.delta.stats.localCache.maxNumFiles", "1")
spark.conf.set("spark.databricks.delta.fastQueryPath.dataskipping.checkpointCache.enabled", "false")
spark.conf.set("spark.databricks.io.cache.enabled", "false")
com.databricks.sql.transaction.tahoe.DeltaLog.clearCache()
spark.sql("CLEAR CACHE")
sqlContext.clearCache()
}
Please find the above piece of custom method to clear all the cache in the cluster without restarting . This will clear the cache by invoking the method given below.
%scala clearAllCaching()
The cache can be validated in the SPARK UI -> storage tab in the cluster.
โ03-29-2022 12:53 PM
Great. You can add โ spark.catalog.clearCache() as well.
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
Click here to register and join today!
Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.