Streaming foreachBatch _jdf jvm attribute not supported

diego_poggioli
Contributor

‎06-17-2024 01:52 AM - edited ‎06-17-2024 02:00 AM

I'm trying to perform a merge inside a streaming foreachbatch using the command:

 
microBatchDF._jdf.sparkSession().sql(self.merge_query)

Streaming runs fine if I use a Personal cluster while if I use a Shared cluster streaming fails with the following error:

org.apache.spark.api.python.PythonException: Found error inside foreachBatch Python process: Traceback (most recent call last):

pyspark.errors.exceptions.base.PySparkAttributeError: [JVM_ATTRIBUTE_NOT_SUPPORTED] Attribute `_jdf` is not supported in Spark Connect as it depends on the JVM. If you need to use this attribute, do not use Spark Connect when creating your session. Visit https://spark.apache.org/docs/latest/sql-getting-started.html#starting-point-sparksession for creating regular Spark Session in detail.

Any idea?

 

Thanks

 

 

 

 

 

0 Kudos

holly
Databricks Employee
Databricks Employee

‎06-20-2024 07:46 AM

Can you share what runtime your cluster is using? 

This error doesn't surprise me, Unity Catalog Shared clusters have many security limitations, but the list is reducing over time. https://docs.databricks.com/en/compute/access-mode-limitations.html#shared-access-mode-limitations-o...

0 Kudos