I need to read and transform several CSV files and then append them to a single data frame. I am able to do this in databricks using simple for loops, but I would like to speed this up.

Below is the rough structure of my code:

for filepath in all_filepaths: 
    df1 = read_file(filepath)
    df2 = transform(df1)
    df3 = df3.append(df2)

Rather than processing 1 file at a time is there a way to process them in parallel? There are plenty of solutions online but I could only get the following to work in databricks:

with ThreadPoolExecutor(max_workers=20) as pool:
    df3 = pd.concat(pool.map(read_and_transform_df, all_filepaths))

For 153 files, the first approach took 3.35 mins and the second approach took 3.87 mins.

Is there a way to optimize the second approach or an alternative faster approach?

Thanks,

Tanjil