Hello,

I am contacting you because I am having a problem with the performance of my notebooks on databricks.

My notebook is written in python (pypark) in it I read a delta table that I copy to a dataframe and do several transformations and create several columns (I have several different dataframes created for it). For the creation of several coumnas I have created a function in my library that takes the initial dataframe as input and returns the same dataframe with the new column.

I can't avoid using the functions but the execution time of my notebook has increased considerably.

I have read that they are executed on the driver node of a Spark application, rather than on the worker nodes where data is distributed and this is the reason why they are slower.

This is the configuration of my cluster:

The driver node is quite large and I have also added the configuration: spark.databricks.io.cache.enabled true.

I am working with approximately 15 million records.

Is there anything I can do to improve this performance? Any additional settings I'm missing that would help my notebook run faster? Right now it is taking an hour and a half.

Thank you so much in advanced.