Hi @Brad ,
It is not clear which cache storage is helping with running your query faster, so the most straightforward way is to reset the sparkContext. Alternatively, these are the three clear cache ways I can think from the top of my head:
// Clear all persistent RDDs from memory, you can verify its effectiveness by monitoring the Storage Tab in the Spark UI
spark.sparkContext.getPersistentRDDs.values.foreach(_.unpersist())
// Disable Databricks IO cache as you are currently doing.
spark.conf.set("spark.databricks.io.cache.enabled", false)
// Clear any cached tables or views if that is what its helping
spark.catalog.clearCache()