I managed to resolve. It was java and python incompatibility with the Spark version I was using.

I will create a video, explaining how to use Spark without Jupyter notebook.

image 

image 

image.png

View solution in original post