How to use Databricks Spark with Ibis

[ccbeloy]

Ibis now includes pyspark backend. However using the databricks-connect doesn't seem to work with Ibis. Here's the sample code the throws an error.

from databricks.connect import DatabricksSession

from databricks.sdk.core import Config

config = Config(profile = "dev")

spark = DatabricksSession.builder.sdkConfig(config).getOrCreate()

df = spark.read.parquet("/somefile.parquet")

df.createOrReplaceTempView("sometable")

import ibis

from ibis import _

ibis_con = ibis.pyspark.connect(spark)

above throws an error:

Python39\site-packages\pyspark\sql\connect\session.py", line 532, in sparkContext

  raise NotImplementedError("sparkContext() is not implemented.")

NotImplementedError: sparkContext() is not implemented.

[jose_gonzalez]

which version of databricks-connect are you using? which DBR version? have you try the notebooks?

[ccbeloy]

Name: databricks-connect

Version: 13.1.0

Databricks Runtime Version

13.1 (includes Apache Spark 3.4.0, Scala 2.12)

No I have not tried the notebook, but was able to successfully use databrick-connect on my local python env (like; connecting to remote cluster and reading parquet file from dbfs).

Thanks.

[Anonymous]

Hi @Calvin Beloy​ 

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. 

We'd love to hear from you.

Thanks!

[ccbeloy]

Hi Khanna,

No solution provided yet. Can escalate with your technical team for a suitable solution please?

Thanks,

Calvin