Hi @desertstorm , The error "The spark driver has stopped unexpectedly and is restarting. Your notebook will be automatically reattached." usually happens when the driver is under memory pressure. This means that there is a piece of code that is executing on the driver and not on executors. We need to identify and remove that piece if code. 

Here are some general suggestions to watch out:-

1. If your code has display or collect operations, you should remove that.

2. If your code has python piece of code, you need to replace that with pyspark.