The Driver was the bottleneck in the Thread Pool approach. By moving to Serverless Workflows, you can shift the orchestration weight to the Databricks Control Plane.

• Eliminate Driver Saturation: Serverless compute for Workflows natively handles task distribution. Databricks provisions the necessary resources for each iteration of the objects automatically.
• For Each Task with Near Instant Scaling: Unlike classic clusters that take minutes to resize, Serverless Performance Optimized mode starts in seconds and uses warm pools to scale concurrent "For Each" iterations (concurrently) without fighting for shared Driver CPU.
• Cost Isolation: Each ingestion is billed for its specific execution time. You avoid paying for an oversized Driver or idle Executors between spikes.