Ibis now includes pyspark backend. However using the databricks-connect doesn't seem to work with Ibis. Here's the sample code the throws an error.

from databricks.connect import DatabricksSession

from databricks.sdk.core import Config

config = Config(profile = "dev")

spark = DatabricksSession.builder.sdkConfig(config).getOrCreate()

df = spark.read.parquet("/somefile.parquet")

df.createOrReplaceTempView("sometable")

import ibis

from ibis import _

ibis_con = ibis.pyspark.connect(spark)

above throws an error:

Python39\site-packages\pyspark\sql\connect\session.py", line 532, in sparkContext

  raise NotImplementedError("sparkContext() is not implemented.")

NotImplementedError: sparkContext() is not implemented.