Update 2: I found the reason in the documentation. This is documented under "Access Mode", and it is a limitation of the Shared access mode.

Link: https://learn.microsoft.com/en-us/azure/databricks/compute/access-mode-limitations#spark-api-limitat...

To quote: " Spark Context (sc),spark.sparkContext, and sqlContext are not supported for Scala in any Databricks Runtime and are not supported for Python in Databricks Runtime 14.0 and above. "