py4j.protocol.Py4JJavaError: An error occurred while calling o359.sql. : java.util.NoSuchElementExce

Data Engineering

Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.

Hi!
We are creating table in streaming job every micro-batch using spark.sql('create or replace table ... using delta as ...') command. This query includes combining data from multiple tables.

Sometimes our job fails with error:

py4j.Py4JException: An exception was raised by the Python Proxy. Return Message: Traceback (most recent call last):
py4j.protocol.Py4JJavaError: An error occurred while calling o359.sql.
: java.util.NoSuchElementException: key not found: Filter (isnotnull(uuid#42326735) AND isnotnull(actor_uuid#42326740))

How can I stop getting this error or add try/except statement to handle it? I was trying this, but it doesn't seem to work and the job continue failing:

from py4j.protocol import Py4JError, Py4JJavaError




try:
    spark.sql.(query)
except Exception as e:
    ## some code
except Py4JError as e:
    ## some code
except Py4JJavaError as e:
    ## some code