What makes the difference is whether the cluster is using Spark Connect or not.
Shared clusters are using Spark Connect, so even the spark session is of different type:

To compare on single user cluster:

What I tested is that you can disable Spark Connect on the cluster by setting spark.databricks.service.server.enabled to false, but in this case everything stops working: