Databricks Worker node - Would like to know number of memory in each core

[Prashanth24]

Under Databricks Compute and Worker nodes, we find different types of types as below

Standard_D4ds_v5 => 16 GB Memory, 4 Cores
Standard_D8ds_v5 => 32 GB Memory, 8 Cores

In Databricks, each node will have one executor. I have questions below

(1) How much memory will be allocated for each core?
(2) For any background process, will be any number of cores and its memory will be allocated?
(3) If there is any backrgound process happens then what are all those activities?

[Rishabh-Pandey]

How much memory will be allocated for each core?

In Databricks, the allocation of memory to each core can be calculated as follows:

• Standard_D4ds_v5:

  • Memory: 16 GB
  • Cores: 4
  • Memory per Core: 16 GB / 4 cores = 4 GB per core

• Standard_D8ds_v5:

  • Memory: 32 GB
  • Cores: 8
  • Memory per Core: 32 GB / 8 cores = 4 GB per core

Thus, each core gets 4 GB of memory in both types of nodes.

[Rishabh-Pandey]

2-For any background process, will there be any number of cores and its memory allocated?

Yes, background processes in Databricks also utilize resources, but their impact on core and memory allocation depends on the workload and the specific processes running. Some common background processes and their resource usage include:

• Driver and Executor Management: The Databricks environment handles the management of the driver and executor processes which run in the background. These processes are allocated cores and memory as needed based on the workload and cluster configuration.
• Job Scheduling and Resource Management: Databricks handles job scheduling and resource allocation, which involves some background processes to ensure efficient resource utilization.
• Monitoring and Logging: Background processes for monitoring cluster performance and logging system metrics use a portion of the available resources.

[Prashanth24]

Thanks for the information. So for this background processing, core and memory from each node will be allocated OR collectively from cluster will be allocated? Also how much core and memory might get allocated for this work per node or per cluster?

[Rishabh-Pandey]

3. If there is any background process, what are all those activities?

Background processes in Databricks include several key activities:

• Cluster Management: Databricks manages the cluster's lifecycle, including starting, stopping, and scaling up or down based on workload demands.
• Job Scheduling: Background processes handle the scheduling and execution of jobs, ensuring that tasks are assigned to the appropriate executors and managed efficiently.
• Resource Allocation: Resources are dynamically allocated and deallocated based on the workload. This includes managing the distribution of cores and memory among various processes.
• Data Shuffling: During data processing, there may be background tasks related to data shuffling and redistribution among different nodes to ensure efficient data processing.
• Error Handling and Recovery: Databricks monitors for errors and failures, automatically handling recovery and reallocation of resources as needed.