cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Maximum VM usage

Sumitsn85
New Contributor

Hi,

I am trying to find what should be the maximum VM's i should reserve as capacity.

I use F8 Cluster instance pool. Multiple jobs use this instance pool during the day.

Most of them overlap during different times of the day.

For reserving capacity, i would like to know what is the maximum Number of VM i am using at any point of the day.

  • Vm
2 REPLIES 2

Anonymous
Not applicable

@Sumit Salianโ€‹ :

Cluster instance pool, you will need to gather data on the usage patterns of your jobs over time.

One way to approach this is to monitor your instance pool's usage and record the number of VMs that are running at each time interval. You can then analyze this data to determine the peak usage periods and calculate the maximum number of VMs that were running during those periods.

To do this, you can use monitoring tools provided by your cloud provider or third-party tools that specialize in tracking cloud usage. These tools can help you collect data on CPU and memory utilization, network traffic, and other metrics that can help you understand how your jobs are using the VMs in your instance pool.

Once you have collected this data, you can analyze it to determine the maximum number of VMs that are needed to handle your workload during peak usage periods. You may also want to consider adding a buffer to your capacity reservation to account for unexpected spikes in demand.

Keep in mind that your usage patterns may change over time, so it's important to continue monitoring and analyzing your instance pool's usage to ensure that you are always reserving the appropriate amount of capacity.

Anonymous
Not applicable

Hi @Sumit Salianโ€‹ 

Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.

Please help us select the best solution by clicking on "Select As Best" if it does.

Your feedback will help us ensure that we are providing the best possible service to you. Thank you!

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group