I have a dataframe with abt 2 million text rows (1gb). I partition it into about 700 parititons as thats the no of cores available on my cluster exceutors. I run the transformations extracting medical information and then write the results in parquet on S3. The process runs for 3 hrs and then crashes. The driver crashed with following error.

The spark driver has stopped unexpectedly and is restarting. Your notebook will be automatically reattached.

I have tried driver with both 128gb and 256 memory but end up in same result. Also i have used the persist option with similar crashes.