- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-07-2024 12:01 AM
@jainshasha base on information you have provided my assumption will be that you might be waiting for Cloud Provider (AWS) to provision VMs (Clusters) for you.
Finding instances for new nodes means that Databricks is attempting to provision the AWS instances necessary. This will often take longer if A) the cluster is larger, or B) the cluster is a spot cluster C) Instance size is on high demand
I don't have AWS Databricks but you can find information if you are using spot instances somewhere in Cluster configuration, there is old article with old UI, but maybe it will help you to find info about if you are using spot or not:
https://www.databricks.com/blog/2016/10/25/running-apache-spark-clusters-with-spot-instances-in-data...
Of course if there is no strong dependency somewhere inside the code that one task is blocking another.
One more thing - because i am more from Azure than AWS - in Azure there is something like QUOTA on Azure Subscription that limits you how many VMs with certain size you can provision at one time.
Maybe there is something like that in AWS that prevents you from starting more than "X" number of clusters at once.
Just advice below:
You can also change your workflow to have one job with multiple tasks running in parallel and configure job cluster for one or for many tasks (reuse cluster), you can save $$ because Databricks will run in parallel as many tasks as cluster can handle and you don't wait for cluster start time (you can provision bigger cluster an let it run in parallel) .
Sorry @jainshasha - i have no more ideas what may be the reason 😞