Hi @MikeGo ,

It is not clear which cache storage is helping with running your query faster, so the most straightforward way is to reset the sparkContext. Alternatively, these are the three clear cache ways I can think from the top of my head:

// Clear all persistent RDDs from memory, you can verify its effectiveness by monitoring the Storage Tab in the Spark UI

spark.sparkContext.getPersistentRDDs.values.foreach(_.unpersist())

// Disable Databricks IO cache as you are currently doing.

spark.conf.set("spark.databricks.io.cache.enabled", false)

// Clear any cached tables or views if that is what its helping

spark.catalog.clearCache()