Databricks Community

olivier-soucy

I'm trying to use the foreachBatch method of a Spark Streaming DataFrame with databricks-connect. Given that spark connect supported was added to `foreachBatch` in 3.5.0, I was expecting this to work.Configuration:- DBR 15.4 (Spark 3.5.0)- databrick...

olivier-soucy

Hello!I'm trying to use the foreachBatch method of a Spark Streaming DataFrame with databricks-connect. Given that spark connect supported was added to `foreachBatch` in 3.5.0, I was expecting this to work.Configuration:- DBR 15.4 (Spark 3.5.0)- dat...

olivier-soucy

From I can remember, I think it was!

olivier-soucy

It turned out that my local machine had a different python version when compared to the workers. Updating python solved this issue. I simply had to look into the driver log, the error message was very obvious: pyspark.errors.exceptions.base.PySparkRu...

olivier-soucy

It's a single user UC enabled cluster.

olivier-soucy

Good call! I should have thought of doing that myself. It turned out that my local machine had a different python version when compared to the workers. Updating python solved this issue. Thank you so much for the help!

olivier-soucy

Hi Daniel!Thanks for getting back to me!Yes it DBR 15.4. It's a single user access mode if that makes a difference. Here is a screenshot of the configuration:And here are the exact version of the pyton packages I'm using:pyspark version: 3.5.0 grpci...

Databricks Community

User Stats

User Activity

Spark Streaming foreachBatch with Databricks connect

Spark Structured Streaming foreachBatch with databricks-connect

Re: Spark Structured Streaming foreachBatch with databricks-connect

Re: Spark Structured Streaming foreachBatch with databricks-connect

Re: Spark Structured Streaming foreachBatch with databricks-connect

Re: Spark Streaming foreachBatch with Databricks connect

Re: Spark Streaming foreachBatch with Databricks connect