%sh runs a shell command on the driver node’s OS, not inside the notebook’s Python/Spark runtime. It basically opens a separate Linux process on the driver machine.

The Spark session, on the other hand, is attached to the notebook runtime. So when you use normal Python cells, you’re inside the Spark-enabled environment.

May I know what you are running using %sh?