topic Re: Spark context not implemented Error when using Databricks connect in Machine Learning

Spark context not implemented Error when using Databricks connect

MightyMasdo — Fri, 07 Jun 2024 09:23:20 GMT

I am developing an application using databricks connect and when I try to use VectorAssembler I get the Error sc is not none Assertion Error. is there a workaround for this ?

Re: Spark context not implemented Error when using Databricks connect

Yeshwanth — Fri, 07 Jun 2024 13:35:08 GMT

@MightyMasdo could you please share the screenshot of the error along with the command?

Re: Spark context not implemented Error when using Databricks connect

Łukasz1 — Tue, 28 Jan 2025 08:02:36 GMT

I have exactly the same problem.

The error is in the line 84 of the file pyspark/ml/wrapper.py.
assert sc is not None

I create spark session with databricks connect as the following:

from databricks.connect import DatabricksSession

spark = DatabricksSession.builder.remote().getOrCreate()

The call stack is the following:

My file: VectorAssembler(inputCols=..., outputCol=...)

pyspark/__init__.py, line 120, in wrapper
return func(self, **kwargs)

pyspark/ml/feature.py, line 5364, in __init__
self._java_obj = self._new_java_obj("org.apache.spark.ml.feature.VectorAssembler", self.uid)

pyspark/ml/wrapper.py, line 84, in _new_java_obj
assert sc is not None

pyspark version is 3.5.0
databricks-connect version is 15.4.4

Re: Spark context not implemented Error when using Databricks connect

pibe1 — Thu, 16 Oct 2025 18:27:02 GMT

Ran into exactly the same issue as @Łukasz1

After some googling, I found this SO post explaining the issue: later versions of databricks connect no longer support the SparkContext API. Our code is failing because the underlying library is trying to fetch the SparkContext (sc)

The databricks docs mention this as well.

I haven't worked out a workaround though.