The executor heartbeat timeout makes sense to investigate first, especially with a table that has grown to around 1.5 crore rows. I’d be interested to know whether increasing executor resources actually resolved the issue before changing the command ...
This is a useful distinction between Python-level threading and Spark’s distributed execution. If the notebooks are creating large Pandas DataFrames on the driver, increasing the thread count could easily make the memory problem worse. Using separate...
