topic Re: How to use Databricks Spark with Ibis in missing-QuestionPost

How to use Databricks Spark with Ibis

ccbeloy — Thu, 08 Jun 2023 19:00:21 GMT

Ibis now includes pyspark backend. However using the databricks-connect doesn't seem to work with Ibis. Here's the sample code the throws an error.

from databricks.connect import DatabricksSession

from databricks.sdk.core import Config

config = Config(profile = "dev")

spark = DatabricksSession.builder.sdkConfig(config).getOrCreate()

df = spark.read.parquet("/somefile.parquet")

df.createOrReplaceTempView("sometable")

import ibis

from ibis import _

ibis_con = ibis.pyspark.connect(spark)

above throws an error:

Python39\site-packages\pyspark\sql\connect\session.py", line 532, in sparkContext

raise NotImplementedError("sparkContext() is not implemented.")

NotImplementedError: sparkContext() is not implemented.

Re: How to use Databricks Spark with Ibis

jose_gonzalez — Wed, 14 Jun 2023 20:59:57 GMT

which version of databricks-connect are you using? which DBR version? have you try the notebooks?

Re: How to use Databricks Spark with Ibis

ccbeloy — Wed, 14 Jun 2023 21:45:38 GMT

Name: databricks-connect

Version: 13.1.0

Databricks Runtime Version

13.1 (includes Apache Spark 3.4.0, Scala 2.12)

No I have not tried the notebook, but was able to successfully use databrick-connect on my local python env (like; connecting to remote cluster and reading parquet file from dbfs).

Thanks.

Re: How to use Databricks Spark with Ibis

Anonymous — Sun, 18 Jun 2023 12:31:56 GMT

Hi @Calvin Beloy

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.

We'd love to hear from you.

Thanks!

Re: How to use Databricks Spark with Ibis

ccbeloy — Tue, 20 Jun 2023 16:36:21 GMT

Hi Khanna,

No solution provided yet. Can escalate with your technical team for a suitable solution please?

Thanks,

Calvin