Retired_mod
Esteemed Contributor III

Hi @a-sky, This message indicates that Databricks is figuring out which file fragments are cached, which can be slow, especially with frequent cluster scaling. To address this, you can try disabling delta caching with `spark.conf.set("spark.databricks.io.cache.enabled", "false")`, optimizing your tables with `OPTIMIZE [table_name]`, upgrading to a newer Databricks Runtime (DBR) version, stabilizing your cluster scaling, and reviewing your job configuration. Let me know if you need any more help or if there’s anything else you’d like to discuss!