cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Maximum VM usage

Sumitsn85
New Contributor

Hi,

I am trying to find what should be the maximum VM's i should reserve as capacity.

I use F8 Cluster instance pool. Multiple jobs use this instance pool during the day.

Most of them overlap during different times of the day.

For reserving capacity, i would like to know what is the maximum Number of VM i am using at any point of the day.

  • Vm
2 REPLIES 2

Anonymous
Not applicable

@Sumit Salian​ :

Cluster instance pool, you will need to gather data on the usage patterns of your jobs over time.

One way to approach this is to monitor your instance pool's usage and record the number of VMs that are running at each time interval. You can then analyze this data to determine the peak usage periods and calculate the maximum number of VMs that were running during those periods.

To do this, you can use monitoring tools provided by your cloud provider or third-party tools that specialize in tracking cloud usage. These tools can help you collect data on CPU and memory utilization, network traffic, and other metrics that can help you understand how your jobs are using the VMs in your instance pool.

Once you have collected this data, you can analyze it to determine the maximum number of VMs that are needed to handle your workload during peak usage periods. You may also want to consider adding a buffer to your capacity reservation to account for unexpected spikes in demand.

Keep in mind that your usage patterns may change over time, so it's important to continue monitoring and analyzing your instance pool's usage to ensure that you are always reserving the appropriate amount of capacity.

Anonymous
Not applicable

Hi @Sumit Salian​ 

Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.

Please help us select the best solution by clicking on "Select As Best" if it does.

Your feedback will help us ensure that we are providing the best possible service to you. Thank you!

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.