topic Re: Databricks Worker node - Would like to know number of memory in each core in Data Engineering

Databricks Worker node - Would like to know number of memory in each core

Prashanth24 — Tue, 13 Aug 2024 12:25:09 GMT

Under Databricks Compute and Worker nodes, we find different types of types as below

Standard_D4ds_v5 => 16 GB Memory, 4 Cores
Standard_D8ds_v5 => 32 GB Memory, 8 Cores

In Databricks, each node will have one executor. I have questions below

(1) How much memory will be allocated for each core?
(2) For any background process, will be any number of cores and its memory will be allocated?
(3) If there is any backrgound process happens then what are all those activities?

Re: Databricks Worker node - Would like to know number of memory in each core

Rishabh-Pandey — Tue, 13 Aug 2024 12:29:54 GMT

How much memory will be allocated for each core?

In Databricks, the allocation of memory to each core can be calculated as follows:

Standard_D4ds_v5:
- Memory: 16 GB
- Cores: 4
- Memory per Core: 16 GB / 4 cores = 4 GB per core
Standard_D8ds_v5:
- Memory: 32 GB
- Cores: 8
- Memory per Core: 32 GB / 8 cores = 4 GB per core

Thus, each core gets 4 GB of memory in both types of nodes.

Re: Databricks Worker node - Would like to know number of memory in each core

Rishabh-Pandey — Tue, 13 Aug 2024 12:30:46 GMT

2-For any background process, will there be any number of cores and its memory allocated?

Yes, background processes in Databricks also utilize resources, but their impact on core and memory allocation depends on the workload and the specific processes running. Some common background processes and their resource usage include:

Driver and Executor Management: The Databricks environment handles the management of the driver and executor processes which run in the background. These processes are allocated cores and memory as needed based on the workload and cluster configuration.
Job Scheduling and Resource Management: Databricks handles job scheduling and resource allocation, which involves some background processes to ensure efficient resource utilization.
Monitoring and Logging: Background processes for monitoring cluster performance and logging system metrics use a portion of the available resources.

Re: Databricks Worker node - Would like to know number of memory in each core

Rishabh-Pandey — Tue, 13 Aug 2024 12:31:12 GMT

3. If there is any background process, what are all those activities?

Background processes in Databricks include several key activities:

Cluster Management: Databricks manages the cluster's lifecycle, including starting, stopping, and scaling up or down based on workload demands.
Job Scheduling: Background processes handle the scheduling and execution of jobs, ensuring that tasks are assigned to the appropriate executors and managed efficiently.
Resource Allocation: Resources are dynamically allocated and deallocated based on the workload. This includes managing the distribution of cores and memory among various processes.
Data Shuffling: During data processing, there may be background tasks related to data shuffling and redistribution among different nodes to ensure efficient data processing.
Error Handling and Recovery: Databricks monitors for errors and failures, automatically handling recovery and reallocation of resources as needed.

Re: Databricks Worker node - Would like to know number of memory in each core

Prashanth24 — Tue, 13 Aug 2024 15:43:26 GMT

Thanks for the information. So for this background processing, core and memory from each node will be allocated OR collectively from cluster will be allocated? Also how much core and memory might get allocated for this work per node or per cluster?

Re: Databricks Worker node - Would like to know number of memory in each core

Retired_mod — Wed, 14 Aug 2024 08:03:20 GMT

Hi @Prashanth24, Thanks for reaching out! Please review the responses and let us know which best addresses your question. Your feedback is valuable to us and the community.

If the response resolves your issue, kindly mark it as the accepted solution. This will help close the thread and assist others with similar queries.

We appreciate your participation and are here if you need further assistance!