Wanted to share something that might save others time and money. We had a complex Databricks script that ran over 1.5 hours, when the target was under 20 minutes. Initially tried scaling up the cluster, but real progress came from simply upgrading the Databricks Runtime to version 16 — the script finished in just 8 minutes, no code changes needed.
Have you seen similar performance gains after a Runtime update? Would love to hear your stories!
I wrote up the details and included log examples in this Medium post (https://medium.com/@protmaks/how-upgrading-to-databricks-runtime-16-4-sped-up-our-python-script-by-1...).
On Medium, anyone can share insightful perspectives, useful knowledge, and life wisdom with the world.
