@MightyMasdo could you please share the screenshot of the error along with the command?

I have exactly the same problem.

The error is in the line 84 of the file pyspark/ml/wrapper.py.
assert sc is not None

I create spark session with databricks connect as the following:

The call stack is the following:

My file: VectorAssembler(inputCols=..., outputCol=...)

pyspark/__init__.py, line 120, in wrapper
return func(self, **kwargs)

pyspark/ml/feature.py, line 5364, in __init__
self._java_obj = self._new_java_obj("org.apache.spark.ml.feature.VectorAssembler", self.uid)

pyspark/ml/wrapper.py, line 84, in _new_java_obj
assert sc is not None

pyspark version is 3.5.0
databricks-connect version is 15.4.4

Ran into exactly the same issue as @Łukasz1 

After some googling, I found this SO post explaining the issue: later versions of databricks connect no longer support the SparkContext API. Our code is failing because the underlying library is trying to fetch the SparkContext (sc)

The databricks docs mention this as well.

I haven't worked out a workaround though.