Today, we start working on setting up an all-purpose cluster pool for all the jobs that we are running on databricks. We used the documentation for this but we got some issues when running our jobs.

The errors in the jobs are the following:

The jobs are running in parallel. To give an explanation:

The pool has the following configuration:

Cluster has the following configuration:

The log4j-active is in the attachment.

Furthermore, I saw that the autoscaling didn't scale up, when doing multiple jobs at the same time. If running multiple jobs on the same pool, it should autoscale right?

Thanks for your time!