3. If there is any background process, what are all those activities?

Background processes in Databricks include several key activities:

• Cluster Management: Databricks manages the cluster's lifecycle, including starting, stopping, and scaling up or down based on workload demands.
• Job Scheduling: Background processes handle the scheduling and execution of jobs, ensuring that tasks are assigned to the appropriate executors and managed efficiently.
• Resource Allocation: Resources are dynamically allocated and deallocated based on the workload. This includes managing the distribution of cores and memory among various processes.
• Data Shuffling: During data processing, there may be background tasks related to data shuffling and redistribution among different nodes to ensure efficient data processing.
• Error Handling and Recovery: Databricks monitors for errors and failures, automatically handling recovery and reallocation of resources as needed.