Thank you @Retired_mod! It looks like the issue was with a recent release on the databricks end, I'd raised a support ticket just to be sure, the difference also died down when we set these two up, at the cluster level.

spark.conf.set("spark.sql.adaptive.enabled", "true")

spark.conf.set("spark.sql.optimizer.excludedRules", "")