DBR-8.3 uses SPARK with version 3.1.x. As per migration guide by default it is restricted to use SparkContext inside the executor. You can enable it by using spark.executor.allowSparkContext

In Spark 3.0 and below, SparkContext can be created in executors. Since Spark 3.1, an exception will be thrown when creating SparkContext in executors. You can allow it by setting the configuration spark.executor.allowSparkContext when creating SparkContext in executors.