I did have the same exact error but I don't have any parallelism especially in that notebook. The failure was in the subsequent notebook for some other parallel runs. like below

1 --> 2,3,4,5 --> 6 --> 7 . This is how my pipeline looks like and 2,3,4,5 are the notebooks running in parallel with almost similar code but different datasets. But the exception occurred on 6. And I didn't have much luck except finding this thread. And to my surprise, this thread is pretty new and I am hoping its not something to do with the DBR version. I am using AWS Databricks 11.3 LTS version with Photon enabled nodes. I am happy to provide any other useful information to troubleshoot this. Thanks