I faced some challenges in my projects related to parallel processing in Databricks. In many cases, the issue was not the volume of data itself, but the overall execution time. I was processing a relatively small number of objects, but each object required separate notebook execution, and the orchestration became the bottleneck.

Current approach

I have a notebook that builds an array of configurations, and in the final step I trigger parallel execution of a general notebook responsible for loading data (for example, Bronze layer ingestion).

Sometimes I process 10 objects, sometimes 60.

My initial solution was based on pool.map():

Where:

• run_notebook_bronze() uses dbutils.notebook.run()
• lst is an array of object configurations

This works quite well for smaller workloads, but I noticed that:

• the driver is utilized close to 100%
• executors are often only at 10% utilization

This clearly shows that the driver becomes the bottleneck, and performance drops significantly for larger workloads.

Three options I found for parallel processing

1. Python multiprocessing / ThreadPool

Pros

• very easy to implement
• fast for small workloads
• simple local testing

Cons

• driver-heavy solution
• poor scalability for larger workloads
• limited executor utilization
• dbutils.notebook.run() overhead becomes significant

2. Creating Jobs dynamically using Databricks Jobs API

Pros

• much better workload distribution
• tasks are managed by Databricks Jobs (information from documentation)

Cons

- don't know

3. Databricks Workflows — For Each task

Then I use these parameters inside a For Each task in Databricks Workflows.

Pros

• probably the best resource and memory management
• native Databricks orchestration
• excellent observability — all runs visible in one place
• production-friendly solution

Cons

• difficult to test manually
• requires running the full Job each time
• cluster startup time can be frustrating during development

My observations

So far:

• multiprocessing is the fastest to implement
• but it performs poorly for heavier workloads because the driver becomes the bottleneck

For Each seems to be the strongest long-term solution, especially for production environments.

Jobs API looks promising as well, but I would like to better understand real production experiences before fully adopting it.

My question to the community

What is your experience with these approaches?

• Which option works best for production-scale parallel notebook execution?
• How do you handle testing for For Each workflows without waiting for cluster startup every time?
• Do you use other patterns for parallel processing in Databricks that provide better memory management and executor utilization?

I would love to hear your perspective, best practices, and lessons learned.