Databricks

dollyb

Hello,I'm using a local Docker Spark 3.5 runtime to test my Databricks Connect code. However I've come across a couple of cases where my code would work in one environment, but not the other.Concrete example, I'm reading data from BigQuery via spark....

dollyb · 03-03-2024

Hi there,what's the best way to differentiate in what environment my Spark session is running? Locally I develop with databricks-connect's DatabricksSession, but that doesn't work when running a workflow job which requires SparkSession.getOrCreate()....

dollyb · 02-21-2024

Hi,I'm using Databricks Connect to run Scala code from IntelliJ on a Databricks single node cluster.Even with the simplest code, I'm experiencing this error:org.apache.spark.SparkException: grpc_shaded.io.grpc.StatusRuntimeException: INTERNAL: org.ap...

dollyb · 03-14-2024

Thanks, dbutils.notebook.getContext does indeed contain information about the job run.

dollyb · 02-23-2024

I just hope Databricks will pay attention to it.

dollyb · 02-23-2024

Yes, I tried both 14.3.0 and 14.3.1.I'm also encountering the same (or very similar) error when firing against a local Spark Connect cluster. When I replace databricks-connect with spark-connect, it works.I sent a bug report to help@databricks.com.

dollyb · 02-23-2024

I also tried on a shared cluster, and the error message is pretty clearorg.sparkproject.io.grpc.StatusRuntimeException: INVALID_ARGUMENT: User defined code is not yet supported.

dollyb · 02-22-2024

So this is clearly a bug in Databricks Connect. I'm not on a support plan, so not sure how to report a bug on this...

Databricks

User Stats

User Activity

Differences between Spark SQL and Databricks

How to detect if running in a workflow job?

Databricks Connect Scala -

Re: How to detect if running in a workflow job?

Re: Databricks Connect Scala -

Re: Databricks Connect Scala -

Re: Databricks Connect Scala -

Re: Databricks Connect Scala -