Clusters are really slow
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-08-2023 11:49 PM
We have two environments for our Azure Databricks. Dev and Prod. We had clusters created and tested in Dev environment, then they were exported to the prod environment through APIs. The clusters in Dev are performing as expected. Whereas, the clusters in Prod are taking a long time for simple select queries. I checked and verified that the clusters have same configuration in both Dev and Prod. What could be causing the issue of slow performance in Prod environment?
- Labels:
-
Azure databricks
-
Cluster
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-09-2023 03:20 AM - edited 11-09-2023 03:25 AM
Both Prod and Dev are connect to unity catalog and I am working with the same table in both the envs. Can something done during the creation of workspace itself affect the performance of clusters? Do clusters update to latest Databricks runtime version automatically? Cause, when I created the clusters I used version 13.2 but the runtime version now is 14.0. I switched it back to 13.2 and performance seems a bit better.
