Databricks Community

dollyb · 04-15-2024

Hello,I'm using a local Docker Spark 3.5 runtime to test my Databricks Connect code. However I've come across a couple of cases where my code would work in one environment, but not the other.Concrete example, I'm reading data from BigQuery via spark....

dollyb · 03-03-2024

Hi there,what's the best way to differentiate in what environment my Spark session is running? Locally I develop with databricks-connect's DatabricksSession, but that doesn't work when running a workflow job which requires SparkSession.getOrCreate()....

dollyb · 02-21-2024

Hi,I'm using Databricks Connect to run Scala code from IntelliJ on a Databricks single node cluster.Even with the simplest code, I'm experiencing this error:org.apache.spark.SparkException: grpc_shaded.io.grpc.StatusRuntimeException: INTERNAL: org.ap...

dollyb · 06-21-2024

Thanks, using the FQN works. I've now added a cluster init-script that removes the (outdated) version provided by Databricks.

dollyb · 06-21-2024

Struggling with this as well. So using dbfs:/ with CREATE TABLE statement works on AWS, but not Azure?

dollyb · 03-14-2024

Thanks, dbutils.notebook.getContext does indeed contain information about the job run.

dollyb · 02-23-2024

I just hope Databricks will pay attention to it.

dollyb · 02-23-2024

Yes, I tried both 14.3.0 and 14.3.1.I'm also encountering the same (or very similar) error when firing against a local Spark Connect cluster. When I replace databricks-connect with spark-connect, it works.I sent a bug report to help@databricks.com.

Databricks Community

User Stats

User Activity

Differences between Spark SQL and Databricks

How to detect if running in a workflow job?

Databricks Connect Scala -

Re: Differences between Spark SQL and Databricks

Re: Cannot get past Query Data tutorial for Azure Databricks

Re: How to detect if running in a workflow job?

Re: Databricks Connect Scala -

Re: Databricks Connect Scala -