Hello all,

I have a large number of light notebooks to run so I am taking the concurrent approach launching notebook runs with dbutils.notebook.run in parallel.

The more I increase parallelism the more I see the duration of each notebook increasing.

I observe that the duration of the cell that includes the imports increases with parallelism up to 20-30 secs:

import org.apache.spark.sql.functions.{col, lit, to_date, date_format}
import org.apache.spark.sql.types.{DateType, TimestampType, IntegerType}
import org.apache.spark.sql.{DataFrame, Row}
import org.apache.spark.sql.catalyst.analysis.NoSuchTableException

Same problem with the cell containing the implicits import statement

import spark.implicits._

FYI I am using a parallelism at max the half of the available drive cores (eg 16 parallelism for an F32s driver)

Is there any strategy that can tackle this issue?

Thank you in advance,