Thank you for the suggestions.

Yes, I have about 60 million rows of data. I found that when I comment out the last line "display(df_final_bs_result)", the code runs great. So it seems like the error could have been triggered by this operation since it attempts to collect and materialize the entire dataframe. I'll try to implement caching to see if that is a workaround as well