Hi, unfortunately I do not have a solution. The solution would be to connect the dataset to an existing spark cluster. It seems that I had spark just locally and all the computations were done locally and that is why the kernel was failing.